We develop and test an incentive-compatible Conjoint Poker (CP) game. The preference data collected in thecontext of this game are comparable to incentive-compatible choice-based conjoint (CBC) analysis data.
We develop a statistical efficiency measure and an algorithm to construct efficient CP designs. We compareincentive-compatible CP to incentive-compatible CBC in a series of three experiments (one online study and twoeye-tracking studies). Our results suggest that CP induces respondents to consider more of the profile-relatedinformation presented to them compared with CBC.
Key words : conjoint analysis; product; measurement and inference; experimental economicsHistory : Received: August 6, 2009; accepted: July 8, 2011; Eric Bradlow and then Preyas Desai served as the
editor-in-chief and Robert Meyer served as associate editor for this article. Published online in Articles inAdvance December 20, 2011.
1. IntroductionConjoint analysis is one of the most widely usedquantitative market research methods (Bradlow 2005,Green and Srinivasan 1990, Wittink and Cattin 1989).By estimating how much a sample of consumersvalue a specific set of features, this method allowsforecasting how these consumers (and, by approxi-mation, the entire market) are likely to react to anyhypothetical set of new products. This enables prod-uct developers to optimize the design and the pric-ing of single products or of entire product lines (e.g.,Kohli and Sukumar 1990, Mahajan et al. 1982).
However, one key limitation of preference measure-ment methods such as conjoint analysis is the poten-tial lack of motivation experienced by respondents(e.g., Ding et al. 2005, Liechty et al. 2005, Netzer et al.2008). There is a growing concern that the amount ofeffort and attention spent by consumers when fillingout preference measurement questionnaires is lowerthan when making real-life purchasing decisions,and practitioners have called for preference mea-surement methods that increase respondents’ level ofinvolvement. For example, Johnson (2008, p. 4) writes,“Although respondents do seem to use simplificationstrategies when filling out questionnaires, they prob-ably work harder when making important real-life
choices. So simplification in answering questions isnot a good thing. We learn less than we might ifwe pushed respondents to use deeper processing. Weneed to find ways to do that.”
One of the significant recent contributions to thepreference measurement literature has been the intro-duction of incentive-compatible mechanisms, whichoffer additional motivation to respondents to pro-vide truthful input. For example, Ding et al. (2005)showed that the external validity of choice-based con-joint (CBC) analysis is dramatically increased whenthe responses given by consumers have an impact ontheir compensation. In particular, the authors askedconsumers to make a series of choices, such that eachrespondent had a positive probability of receivinghis or her preferred alternative from each choice setafter the end of the experiment. Ding (2007) extendedthis method to situations in which the researcherhas access to only a few alternative products andtherefore is unable to offer all the alternatives in allthe choice sets as potential rewards. Whereas Ding(2007) required estimating how much each respon-dent would be willing to pay for each potentialreward product (based on that respondent’s choices),Dong et al. (2010) showed that similar predictiveperformance may be achieved by simply inferring
the respondent’s rank ordering of the set of possiblereward products.
Researchers have also recently developed incentive-compatible preference measurement tasks that deviatefurther from traditional conjoint analysis. For exam-ple, Ding et al. (2009) proposed an online incentive-compatible preference measurement mechanisminspired by barter markets. Park et al. (2008) proposeda mechanism that relies on allowing participants toupgrade products, where incentive-compatibility isachieved by using the Becker–De Groot–Marschak(BDM) procedure (i.e., the participant states his orher willingness to pay for an upgrade, a randomprice is generated, and the transaction is realized atthat price if and only if it is lower than or equal tothe stated willingness to pay).
The objective of incentive compatibility is to inducetruth telling, which requires participant involvementand attention. (Note that involvement and attentionare not sufficient conditions for truth telling.) How-ever, current incentive compatible preference mea-surement methods may not increase involvement tothe level of real-life purchasing decisions. For exam-ple, consider a consumer making a decision on whichlaptop computer to purchase with his or her ownmoney. He or she may pay more attention to the infor-mation relevant to that decision compared with a typ-ical CBC context in which he or she is asked to makemany consecutive choices between laptop computers,where each choice has only a probabilistic link to areal outcome and where this outcome involves prizemoney (as opposed to his or her own money).
In this paper we explore additional ways to increaseinvolvement and attention in preference measure-ment while maintaining incentive compatibility. Wedevelop and test an incentive-compatible “ConjointPoker” (CP) game inspired by regular poker. In thisgame, each card represents a product defined by acombination of features. “Hands” are defined simi-larly to regular poker (e.g., a pair is a hand in whichtwo products have one feature in common). The pref-erence data revealed by respondents during this gameare comparable to incentive-compatible CBC data. Wedevelop a statistical efficiency measure and an algo-rithm to construct optimal CP designs. We compareincentive-compatible CP to incentive-compatible CBCin a series of three experiments. Our first study, abetween-subjects online experiment, provides indirectevidence that CP participants consider more of theprofile-related information presented to them com-pared with CBC participants. We then conduct twoeye-tracking studies that provide convergent, directevidence for this effect.
The rest of this paper is structured as follows. In§2 we introduce CP, develop a model to analyze datafrom the game, a measure of statistical efficiency for
CP designs and an algorithm for constructing efficientCP designs. We report the results of our experimentsin §3 and conclude in §4.
2. An Incentive-Compatible PokerGame for Preference Measurement
In this section we describe an incentive-compatibleConjoint Poker game. Different versions of the gamemay be developed; we have developed four-card andseven-card versions. The seven-card version, inspiredby Texas Hold’em, is described in the electronic com-panion (available as part of the online version that canbe found at http://mktsci.journal.informs.org/). Wefocus here on the four-card version, which is looselybased on three-card poker.
2.1. OverviewInstead of using traditional playing cards defined bytwo attributes (number and color) with 13 and 4 lev-els, respectively, cards in this game represent prod-uct profiles described by any number A of attributes(A = 6 in our experiments), where each attribute ahas La levels. See Figure 1 for an example of a card.Although our implementation is online, the gamemay also be played off-line using physical cards. Inits online format, the game may be played betweenmultiple consumers or against the computer. Allow-ing for multiple players raises several issues withrespect to design efficiency, estimation, and learningand information diffusion among respondents (Dinget al. 2009). We leave the investigation of multiple-player versions to future research and focus in thispaper on versions of the game in which each respon-dent plays against the computer only.
Step 2. Hand selection stage: Player creates a three-cardhand. (Clicking on a card flips that card. Hand is composedof cards left face up.)
Step 3. Card selection stage: Player indicates his or herpreferred card in the hand. He or she may receive the producton that card (plus the difference between some presetamount of money and the price of that product when price isan attribute) if he or she wins that round. The probabilityof winning is still positive but smaller in the case of a tie.
Step 4. Winner of the round is announced (based on handstrength—ties are allowed).
Step 1. Four cards are presented to the player.
The game is played in rounds. Each round has twostages, a hand selection stage and a card selection stage.See Figure 2 for an illustration of the steps involvedin each round. In the hand selection stage, each playeris asked to form a three-card hand from a set offour cards. As in regular poker, a hand is a set ofcards that have a specific pattern. Hands have dif-ferent strengths, and the winner in each round is theplayer with the strongest hand (ties are allowed). Thestrength of a hand is based on the probability thatthis hand would be achievable from a random set offour cards (drawn without replacement from the setof all cards). Because the number of attributes andlevels in CP do not typically match those in regularpoker (two attributes with 13 and 4 levels, respec-tively), the various types of hands are defined a lit-tle differently, and the probabilities corresponding toeach hand need to be computed. These probabilities
are used to determine the relative strengths of the dif-ferent hands (such that less likely hands have higherstrength) and will also be used in the choice modeldeveloped below. See Appendix A for details on thecomputation of these probabilities. We use the fol-lowing six types of hands, listed from weakest tostrongest (see Figure 3 for an illustration):
—One pair (weakest): two cards have the samelevel on one attribute.
—Straight: all three cards have different levels onone attribute.
—Double straight: all three cards have differentlevels on two attributes.
—Flush: all three cards have the same level on oneattribute.
—Straight flush: all three cards have the same levelon one attribute and different levels on another.
Figure 3 Examples of Hands (from Weakest to Strongest): One Pair (a), Straight (b), Double Straight (c), Flush (d), Straight Flush (e), andDouble Flush (f)
+
(e)
(a)
(c) (d)
(b)
(f)
One card
—Double flush (strongest): all three cards have thesame level on two attributes.
In the card selection stage, each player is asked toindicate his or her preferred card from that hand.1
This information is used to provide incentives torespondents, as described next.
2.2. IncentivesAt the end of the experiment, one player is selectedrandomly, and one of the rounds played by thatplayer is selected randomly. If the player won thatround, then he or she wins the product on his or herpreferred card. If price is one of the attributes, thenthe player also receives the difference between a pre-set amount of money and the price of that product.If the player was tied for best in that round, he orshe receives the reward with a probability equal to1 divided by the number of players in the tie. If the
1 In our implementation, players were asked to choose one card.Other implementations may introduce a “no-choice” option.
player lost that round, then he or she receives noth-ing (except for any potential nominal fee paid to allrespondents). In cases in which the experiment alsoinvolves an external validity task, there is a positiveprobability that the incentives will be based on thattask instead (see the setup of our experiments in §3).2
For the sake of argument, let us assume risk neu-trality in the hand selection stage (this assumptionwill be relaxed in our final model). We define the util-ity of a card as the utility of the product on that cardplus the utility of the difference between the presetamount of money and the price of that product, ifapplicable. Given the incentive structure, the expectedutility derived by a respondent from a given round ifplaying a given hand, h, is then proportional to the
2 This mechanism is based on the “random lottery procedure,”which has been widely used and validated in experimental eco-nomics (Starmer and Sugden 1991). See Ding et al. (2005, 2009) forother applications of this procedure to incentive-compatible con-joint analysis.
probability of winning that round with h, multipliedby the utility of the respondent’s preferred card in h:Expected utility if play hand h ∝ Probability of winningthe round with h× Utility of preferred card in h.
This implies a trade-off. On the one hand, playersshould play strong hands to increase the probabilitythat they will receive a prize. On the other hand, play-ers should also play hands that contain cards that theylike in order to increase the utility from the potentialprize. This implies that the optimal strategy is neitherto always play the strongest hand nor to play a handthat contains the card with the highest utility in eachround. In particular, it may be optimal to play a handthat is not the strongest hand but contains a high-utility card or to play a hand that does not containone’s favorite card but has a higher chance of winningthe round.
The above expression was provided only to illus-trate the basic trade-off faced by CP participants. Inreality, we do not expect all consumers to be risk neu-tral in the hand selection stage. In the next section,we develop a choice model that captures this trade-offwithout assuming risk neutrality.
2.3. Consumer Choice ModelWe now propose a choice model that captures thehand selection and card selection stages of CP. Leti index consumers, r index the rounds in the game,h index the possible hands available to players, andj index the profiles (i.e., cards). When there are fourcards per round and when each hand consists of threecards, each player has a choice between four possi-ble hands in each round. We index these four handsby h ∈ 811213149 and the four cards by j ∈ 811213149,where h = 1 corresponds to the hand made of pro-files 8112139, h = 2 corresponds to profiles 8213149,etc. With a slight abuse of notation, we write j ∈ hif card j is present in hand h. Let hi1 r be the handselected by consumer i in round r in the hand selec-tion stage, and let ji1 r be the profile selected by thatconsumer in the card selection stage among the pro-files in hand hi1 r . To estimate partworths from thechoices made by consumers, we construct a likelihoodfunction for Pr({hi1 r1 ji1 r }). We have
Pr48hi1 r1 ji1 r95= Pr4hi1 r 5× Pr4ji1 r � hi1 r 50 (1)
The second component, Pr(ji1 r � hi1 r 5, corresponds tothe card selection stage in which consumer i choosesone profile from a set and may be modeled simplyusing logistic probabilities:
Pr4j � hi1 r 5=
exp4xj�i5∑
j ′∈hi1 rexp4xj ′�i5
if j ∈ hi1 r1
0 otherwise1
(2)
where �i corresponds to the partworths for con-sumer i, and xj is an appropriately coded row vectorthat captures the attribute levels in profile j .
The other term in Equation (1), Pr(hi1 r 5, correspondsto the hand selection stage, which is unique to CP.Here also, the consumer needs to make a choicebetween a set of possible alternatives (i.e., hands).In the case of risk neutrality, the expected utilityfrom choosing a hand is proportional to product ofthe probability of winning with that hand and theutility of the preferred card in that hand. Note thatwhereas a utility intercept is not identified from thecard selection stage alone in the absence of a “no-choice” option, such an intercept is identified fromthe hand selection stage because of the possibility ofnot winning anything in the round. In particular, ifwe normalize the utility of not winning anything (i.e.,of losing the round) to 0, then the expected utilityobtained by a risk-neutral consumer i from playinghand h is equal to V i1 r
h = Pwh � r 4�i+maxj∈h8xj�i95, where
Pwh � r is the probability of winning round r by play-
ing hand h, and �i is an intercept that captures con-sumer i’s utility from winning a prize in the game. Ina two-player game, Pw
h � r is equal to the probabilitythat the other player’s best hand is strictly weakerthan h plus half of the probability that the otherplayer’s best hand is exactly as good as h (ties are bro-ken randomly). In our experiments, each consumerplayed against the computer, the computer’s cardswere drawn randomly without replacement from theset of all possible cards, and the computer alwaysplayed the strongest hand in each round. The result-ing winning probabilities Pw
h � r are computed in closedform in Appendix A.
The above expression assumed risk neutrality inthe hand selection stage; i.e., the utility of the pre-ferred card in a hand was assumed to have a pro-portional influence on the player’s evaluation of thehand. Risk-averse (respectively, risk-seeking) behav-ior is obtained when that the player’s evaluation ofthe hand is a concave (respectively, convex) functionof the utility of the preferred card in the hand. Mak-ing the standard assumption of constant relative riskaversion gives rise to the following expression:
V i1 rh = Pw
h � r
(
�i + maxj∈h
8xj�i9)�i
1 (3)
where �i is the risk-aversion parameter for con-sumer i. Risk neutrality is obtained when �i = 1,�i < 1 gives rise to risk aversion in hand selection, and�i > 1 gives rise to risk-seeking behavior.
We then model consumer i’s choice in the handselection stage as
where � is a logit scale parameter that capturesthe possibility that the response error is different inthe hand selection and card selection stages (e.g.,because of misrepresentations of the probabilities ofwinning).3
2.4. Design AlgorithmWhen estimating partworths from conjoint analysisdata, the statistical efficiency of the estimates, typi-cally measured by their asymptotic covariance matrix,depends on the profiles shown to consumers. Inthe case of CBC, this asymptotic covariance matrixdepends further on the partworth estimates them-selves (Huber and Zwerina 1996). Under mild con-ditions the asymptotic covariance matrix is equal tothe inverse of the information matrix (see, for exam-ple, McFadden 1974, Newey and McFadden 1994).A series of statistical efficiency measures have beenproposed, typically based on various types of matrixnorms (e.g., determinant, trace) applied to the infor-mation matrix, and various design algorithms havebeen proposed to select profiles that maximize effi-ciency. In CBC, a common efficiency measure isD-efficiency (based on the determinant norm), anda common approach for creating D-efficient designsis to obtain some prior information on the part-worths and, given that information, to apply a setof operators to transform a nonefficient design intoa D-efficient design (see, for example, Huber andZwerina 1996; Sandor and Wedel 2001, 2002, 2005).Four properties characterize D-efficient CBC designs:level balance (the levels of an attribute occur withequal frequency), orthogonality (any two levels of dif-ferent attributes appear in profiles with frequenciesequal to the product of their marginal frequencies),minimal overlap (each attribute level repeats itselfwithin each choice set with minimal probability), andutility balance (profiles in each choice are similarlyattractive).
This design approach may be extended to the con-struction of efficient CP designs. The key difference isin the computation of the information matrix. Givenour choice model above, the log-likelihood function isequal to
L4data �X1�1�1�1�5
=∑
i
∑
r
log4Pr4hi1 r 55
+ log4Pr4ji1 r � hi1 r 55+ constant0
3 An alternative version of the game would be such that players stillwin some amount of money if they lose the round. The same mod-eling framework could be applied to a situation like this by sim-ply replacing V i1 r
h = Pwh � r 4�i + maxj∈h8xj ·�i95
�i with V i1 rh = Pw
h � r 4�i +
maxj∈h8xj ·�i95�i + 41 − Pw
h � r 5 · �i , where �i would capture the utilityfrom winning the amount of money offered when the round is lost.
Taking the Hessian of the likelihood function withrespect to �, we find that the expected value ofthe information matrix ì is proportional to (seeAppendix B for details):4
ì =∑
r
[
∑
hr∈r
z̃rhr Pr4hr 5z̃Thr
+ Pr4hr 5∑
j∈hr
zj �hr Pr4j � hr 5zj �hTr
]
1 (5)
where Pr4hr 5 is given by (4), Pr4j � hr 5 is given by (2),and
z̃rhr = � ·ïV rhr
−∑
h′r
� ·ïV rh′rPr4h′
r 51
zj �hr = xTj −
∑
j ′∈hr
xTj ′ Pr4j ′ � hr 51
ïV rhr
=�Pwhr �r
4�+xj∗hr�5�−1xT
j∗hr1 where j∗hr =argmax
j∈hr
4xj�50
The D-efficiency of any CP design may be computedfrom the determinant of the corresponding informa-tion matrix using the same formulas as the ones usedto compute the D-efficiency of a CBC design givenits information matrix. Moreover, the same opera-tors (e.g., swapping, relabeling) used to improve theD-efficiency of a CBC design may also be used toimprove the D-efficiency of a CP design.
One important difference between CP and CBC isthat efficient CP designs tend to have more leveloverlap compared with efficient CBC designs such asthose considered here (i.e., standard logit designs).5
Minimal overlap occurs when attribute levels arerepeated within a choice set as little as possible. Forexample, in our experiments we achieved no overlapin our CBC design; i.e., each level of each attributeappeared exactly once in each choice set, leading tomaximal design efficiency. In contrast, some overlapis actually desirable in CP designs. Indeed, in a CPdesign with no level overlap, all hands in all roundsare equally strong (all hands are double straights, andPwh � r is constant for all r and h), and the intercept �i
and risk-aversion parameter � i are not identified (allhands have the same probability of winning).
In our field study, we followed Huber and Zwerina(1996) and conducted a pretest from which we
4 The expression in Equation (5) assumes a homogeneous partworthvector � (Huber and Zwerina 1996). In Appendix B, we providethe information matrix for the mixed logit model. This informationmatrix involves integrals and does not have a closed form (Sandorand Wedel 2002). For computational simplicity, the designs usedin our experiments assume a homogeneous partworth vector (i.e.,standard logit model).5 Sandor and Wedel (2002) showed that efficient mixed logit designstend to have more level overlap compared with efficient standardlogit designs.
obtained prior estimates (our pretest had 56 and 58respondents in the CBC and CP conditions, respec-tively). We then applied the relabeling and swappingoperators to improve the D-efficiency of a design thatwas selected as a starting point (see, for example,Huber and Zwerina 1996; Sandor and Wedel 2001,2002, 2005). For CBC, we used a D0-efficient designwith no overlap as a starting point (a D0-efficientis a D-efficient design assuming that all partworthsare equal to 0). This D0-efficient design was obtainedusing the standard cyclic approach of Bunch et al.(1994): starting with an orthogonal design, a set ofchoice alternatives was constructed by adding cycli-cally generated alternatives to each set. For CP, weused a perturbed version of that D0-efficient designas a starting point, where the perturbation was per-formed in order to introduce some amount of leveloverlap.6
3. Experiment DetailsWe now report the results of three experimentsthat compared incentive-compatible CP to incentive-compatible CBC. Our first study was a between-subjects online experiment that enabled us to comparethe partworth estimates across the two methods andprovided indirect evidence that CP participants con-sidered more of the profile-related information pre-sented to them compared with CBC participants.Studies 2 and 3 are within-subjects eye-tracking stud-ies that enabled us to measure more directly theamount of information considered by respondentsunder the two methods.
3.1. SetupThe setup of all three studies was similar. CBC and CPwere implemented on the same online platform withthe same user interface. See Figure 2 for screenshotsof the CP interface and Figure 4 for screenshots of theCBC interface. We used mini laptops as our productcategory. Each of our product profiles was a differ-ent customized version of the Dell Inspiron Mini 11zlaptop computer. The display (1106" high-definition
6 The cyclic approach of Bunch et al. (1994) is such that if attributea is at level l in the qth profile of the orthogonal design, then it isat level l in the first card in the qth question of the choice design,level l+ 1 in the second card, etc. In other words, starting from theorthogonal design, the level of each attribute is incremented by 1for each new card in the choice set. In our perturbed version, it wasincremented with probability 1−t instead of 1. Therefore if attributea was at level l in the qth profile of the orthogonal design, it wasat level l in the first card in the qth round of Conjoint Poker. Thevalue in the second card was then equal to l + 1 with probability1 − t and to l with probability t. The value in the third card wasthen equal to that in the second card plus 1 with probability 1 − tand equal to that in the second card with probability t, etc. We usedt = 00350
WLED display), processor (1 3 GHz Intel® Celeron 743processor), RAM (2 GB), battery (28WHr lithium-ionbattery), operating system (Genuine Windows® VistaHome Edition), and webcam (integrated 1.3 M pixelwebcam) were held constant. Six attributes were var-ied, with four levels each: design (four different colorschemes: promise pink, obsidian black, jade green,or ice blue), warranty (1-year limited, 1-year limitedwith in-home service after remote diagnosis, 2-yearlimited with in-home service after remote diagnosis,or 3-year limited with in-home service after remotediagnosis), McAfee® SecurityCenter antivirus (30-day,15-month, 24-month, or 36-month subscription), harddrive (120 GB, 160 GB, 250 GB, or 320 GB), accessory(Logitech® black cordless mouse, Logitech red cordlessmouse, Linksys wireless router, or Creative Labs head-phones), and price ($500, $550, $600, or $650). Giventhis price range, and given the fact that Dell laptopsare customizable, we were able to offer any productprofile as an incentive (like in Ding et al. 2005).7
In each study, the flow of the experiment was asfollows, for each respondent:
1. Instructions: Detailed instructions (using pictureillustrations) were displayed on the introductory page.Care was taken to make the instructions in both condi-tions as symmetric as possible. In addition, we createda slide show for each condition with a shorter ver-sion of the instructions, which we embedded at the topof the page (using http://www.authorstream.com—slide shows and instructions are available from theauthors upon request). The slide shows in both condi-tions used similar language (e.g., in both conditions,profiles were referred to as “cards”). Throughout theexperiment, a link to the instructions was available toparticipants.
2. External validity task: As an external validity task,participants were asked to select one card from aset of eight. See Figure 5 for a screenshot. The for-mat of the external validity task and the set of eightprofiles were identical across conditions and partic-ipants. These eight profiles were randomly selectedsubject to a level-balancing constraint (each levelof each attribute appears exactly twice across pro-files) and such that exactly two of the four orderedattributes (warranty, security software, hard drive,price) were at one of the two most attractive levels ineach profile. This last constraint was added to avoiddominance.
3. Main task: Participants in the CBC condition weregiven 20 CBC questions, each with four alternatives.Participants in the CP condition were asked to play20 CP rounds against the computer. (CP participantswere given two practice rounds between Steps 2
7 Although appealing, this property is not required. See, for exam-ple, Ding (2007) and Dong et al. (2010).
and 3.) See Figures 2 and 4 for screenshots from eachcondition. The CBC and CP designs were selected tobe D-efficient as explained above. The only exceptionis the second eye-tracking study (Study 3), in which
Figure 5 External Validity Task
both designs were identical. Within each condition,the design was identical across respondents.
4. Follow-up questionnaire: Participants were ad-ministered a short follow-up survey that measured
their feedback on the task, their knowledge of regularpoker, and how much they would need to be paid inorder to participate in a similar study in the future.(More details are provided in §3.2.3.)
3.2. Study 1Our participants were recruited from a commercialonline panel typical of online market research. Par-ticipants were not screened based on their knowl-edge of poker or any other criteria. Respondentsaccessed the experiment from the Internet. We useda between-subjects design with N = 318 in each con-dition. Respondents were given a flat nominal incen-tive to participate (in the form of points that maybe redeemed for rewards), in addition to the fol-lowing incentives. In both conditions participantswere informed (in Step 1) that one participant wouldbe selected randomly at the end of the study andthat one of the tasks (external validity or CBC inthe incentive-compatible CBC condition, or externalvalidity or CP in the incentive-compatible CP con-dition) would be randomly selected for that partici-pant. If the external validity task was to be selected,the winner would receive the product on his or herpreferred card and the difference between $650 andthe price of that product. If the CBC task was to beselected, then one of the 20 CBC questions wouldbe selected, and the winner would receive his or herpreferred card from that question and the differencebetween $650 and the price of that product. If the CPtask was to be selected, then one of the rounds wouldbe randomly selected. If the selected respondent wasto have won this round, then he or she would receivethe product on his or her preferred card in that handand the difference between $650 and the price of thatproduct. If the selected respondent was to have lostthis round, then he or she would receive nothing(with the exception of the flat incentive offered to allrespondents). If the selected respondent was to havebeen tied with the computer in this round, then heor she would receive the prize with probability 0.5.Note that the incentives given to the participants inthe CP condition were smaller in expectation com-pared to the CBC condition, making the comparisonsconservative.
3.2.1. Parameter Estimates. We estimated therespondents’ partworths in each condition using hier-archical Bayes (HB) (Rossi and Allenby 2003). Detailsare provided in Appendix C. In each condition, thefirst stage prior on �i was normal: �i ∼ N4�01D�5.In the CP condition, we also specified priors on�i (risk-aversion parameter) and �i (intercept):�i ∼ TN4�01D� 5 (truncated between 0 and 2 fornumerical stability) and �i ∼ N4�01D�5. Table 1reports the estimates of the population means ofall partworths (i.e., posterior mean of the first-stage
prior parameters) as well as 95% credible intervalsfor these population means and the point estimateof the standard deviation of each parameter acrossconsumers (i.e., the square root of the diagonalterms of the matrix D�5. We used effects codingsuch that the partworths always sum to 0 withinan attribute. We see that the average partworthestimates obtained from the two methods are fairlydifferent. In fact, the 95% credible intervals do notoverlap for 9 out of the 24 attribute levels. Thepartworth estimates based on incentive-compatibleCBC are also more heterogeneous than those basedon incentive-compatible CP. The average standarddeviation (across attribute levels) of the partworthsis equal to 0.748 with incentive-compatible CBC and0.414 with incentive-compatible CP. The partworthsobtained from incentive-compatible CBC are alsomore varied at the individual level. In particular,the average across respondents of the ratio betweenthe importances of the most and the least importantattributes is 20.238 under incentive-compatible CBCcompared with 11.803 under incentive-compatible CP.
Further analysis of the raw data provides an expla-nation for these differences. In particular, a substantialproportion of respondents in the incentive-compatibleCBC condition made all of their CBC choices basedon a very limited subset of attributes and levels. Forexample, 74 of the 318 participants in the incentive-compatible CBC condition selected profiles that allhad one attribute in common (e.g., all the profilesselected across the 20 CBC questions had 320 GB harddrives). Recall that D-efficient CBC designs have min-imal overlap; i.e., each attribute level repeats itselfwithin each choice set with minimal probability. Inour case, a design with no overlap was available; i.e.,in each choice set, each attribute level was presentexactly once. This implies that partworth estimatesfor these respondents are very uneven and are onlyidentified through shrinkage. Consider, for example,a respondent who always selects laptops with 320 GBin all CBC questions. Because there is only one 320 GBlaptop in each question, the choice data from thisrespondent may be fitted perfectly by assigning avery large positive partworth to 320 GB and smallerweights to all other attribute levels, giving rise touneven partworth estimates. Moreover, the only infor-mation contained in this respondent’s data is that320 GB is greatly preferred to all other partworths,but no information is provided on how large thedifference is or on the relative preferences for theother attribute levels.8 Therefore this respondent’s
8 More precisely, as long as the partworth for 320 GB minus thepartworth for the next level of this attribute is greater than the sumover all other attributes of the highest minus the lowest partworths,the respondent would always prefer the laptop with 320 GB, irre-spective of the other attribute levels.
Notes. The population means estimate is reported first. The 95% credible interval of the population means estimateis reported second in brackets, and the point estimate of the standard deviation of the parameter across consumersis reported last.
partworths are identified only because of shrinkage;i.e., without the shrinkage performed by HB, the like-lihood function corresponding to such a respondentwould not have a finite maximum.9
Interestingly, the 74 incentive-compatible CBCrespondents who selected profiles that were alwayssimilar on one attribute may actually be groupedbased on the attribute level that is common acrosstheir preferred profiles. For example, 29 of theserespondents always selected laptops with 320 GBhard drives, 13 always selected laptops with wirelessrouters, and 9 always selected pink laptops. Becausethere was no level overlap in the D-efficient CBCdesign, within each of these groups, all respondentsgave the exact same answers to all CBC questions. Infact, 90 of the 318 incentive-compatible CBC respon-dents (28.30%) gave answers that were completely
9 Johnson (2008, p. 3) also notes the limitations of questionnairedesigns with minimal overlap when respondents “use simplifica-tion strategies, such as always choosing a preferred brand, or thelowest price, or the most extreme level of some other attribute.”
identical to the answers of at least one other respon-dent in that condition. Such a phenomenon was notpresent in the CP data. None of the CP respondentsgave answers that were all similar to the answers ofanother CP respondent, and none of the CP respon-dents selected profiles that all had one attribute incommon.
Although the evidence so far is only indirect, theseobservations suggest that CBC respondents consid-ered less of the profile-related information when mak-ing their choices compared with CP respondents. Thenext two studies confirm this phenomenon by provid-ing direct evidence from eye-tracking data.
Finally, we discuss the estimates of the otherparameters that are unique to CP. Table 2 reportsthe estimates of �, �0, and �0, and Figure 6 reportsthe distribution of the estimates of the risk-aversionparameter �i across CP participants. For each respon-dent, we construct a 95% credible interval for �i0 For3 of the 318 respondents, the upper bound of thiscredible interval is below 1, which corresponds torisk aversion in the hand selection stage. For 93 of
Note. The 95% credible intervals are reported in brackets.
the 318 respondents, this credible interval is above 1,which corresponds to risk seeking in the hand selec-tion stage. Therefore, our results suggest that a signif-icant proportion of CP participants are risk seeking intheir selection of hands. Note that risk-seeking behav-ior is not uncommon in settings such as this one (see,for example, Thaler and Johnson 1990).
3.2.2. Performance Comparisons. As we showlater, our eye-tracking studies suggest that respon-dents consider only a subset of the profile-relatedinformation contained in the external validity task.This raises questions about the ecological validity ofsuch tasks in general and about their ability to detecta deeper level of processing in the calibration task inparticular. Indeed, it is unclear how a deeper level ofprocessing in the calibration task would impact pre-dictive performance in the validation task, if the levelof processing in the validation task itself is not asdeep. With this caveat, in this section we compare thepredictive performance of CBC versus that of CP forcompleteness.
We refer to the respective choice shares of theeight profiles in the external validity task based onthe entire sample (i.e., both conditions combined—636 respondents) as the “observed out-of-sampleaggregate choice shares.” For each method, we com-pute at each posterior draw in the Markov chainMonte Carlo (MCMC) the logistic probability thateach respondent will choose each of the eight profiles
Figure 6 Distribution Across Conjoint Poker Participants of the PointEstimate of the Risk-Aversion Parameter �i
0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.00
5
10
15
20
25
30
�
Num
ber
of p
artic
ipan
ts
in the external validity task. Averaging these proba-bilities across posterior draws gives us point estimatesof each respondent’s choice probabilities in the exter-nal validity task. We use these estimated probabilitiesto compute hit rates (where the prediction is that eachrespondent will choose the profile with the highestpredicted choice probability). We also measure howwell the aggregate predictions made by each methodcoincide with the observed out-of-sample aggregatechoice shares. We compute an estimate of the out-of-sample aggregate choice shares at each posterior drawin the MCMC by averaging the choice probabilitiesacross respondents. Point estimates are obtained byaveraging these estimates across posterior draws. Theaccuracy of these predictions is measured using themean absolute deviation (MAD), the correlation, andthe root mean squared error (RMSE) between the pre-dicted and observed out-of-sample aggregate choiceshares. For all performance metrics, we also report95% credible intervals across posterior draws. We notethat the accuracy of out-of-sample choice share pre-dictions is typically assumed to be highly correlatedwith the accuracy of out-of-sample individual-levelpredictions, and it is therefore often not reported inpapers on CBC. However, we will see that this corre-lation may not be as high as is usually assumed.
The results are reported in Table 3. The hit ratesachieved by incentive-compatible CBC and incentive-compatible CP are not significantly different (i.e., the95% credible intervals overlap10). However, incentive-compatible CP produces out-of-sample aggregatechoice share predictions that are significantly moreaccurate than those produced by incentive-compatibleCBC. In fact, the aggregate choice share predic-tions based on incentive-compatible CBC are worsethan chance.11 Table 4 gives the actual and pre-dicted choice shares for each profile in the externalvalidity set. The fact that incentive-compatible CBCpredicts out-of-sample individual choices well butpredicts out-of-sample aggregate choice shares poorlyis counterintuitive and worthy of subsequent inves-tigation. In particular, future research may test thereplicability and generalizability of this finding andidentify its cause.
3.2.3. Follow-up Questionnaire. After complet-ing the task, subjects in both conditions answered afollow-up questionnaire. The goal of this question-naire was to further test the level of engagement of
10 The difference is not significant either when comparing the twosets of hit rates using a t-test (p-value = 0074).11 The difference between the accuracy of the out-of-sample choiceshare predictions made by the two methods is even more pro-nounced when the accuracy of each method is evaluated based onthe shares observed in that same condition (details available fromthe authors upon request).
Notes. Based on external validity task. A null model predicting equal choice probabilities across eight profiles would achieve a hit rate of 0.125, a MAD of0.0623, and a RMSE of 0.0667 (correlation is not defined when all share predictions are equal). The 95% credible intervals are reported in brackets.
Table 4 Detailed Choice Share Predictions
Profile description Observed Choice share predicted by Choice sharechoice incentive-compatible predicted by
Color Warranty Security software Hard drive Accessory Price ($) share (%) CBC (%) CP (%)
Pink 1 yr in-home 36 mos 320 GB Red mouse 600 17092 32042 19089Green 1 yr limited 30 days 250 GB Headphones 550 3062 11099 6095Blue 2 yrs in-home 36 mos 160 GB Wireless router 650 19065 14034 16065Black 3 yrs in-home 24 mos 160 GB Wireless router 600 16035 14021 17047Pink 3 yrs in-home 24 mos 120 GB Headphones 650 3014 7091 5006Green 1 yr limited 15 mos 320 GB Red mouse 550 19034 10026 21070Blue 2 yrs in-home 15 mos 120 GB Black mouse 500 5082 5031 4060Black 1 yr in-home 30 days 250 GB Black mouse 500 14015 3057 7068
the respondents and to test their knowledge of regu-lar poker.
Engagement. Our results so far (and the eye-trackingstudies reported next) suggest that CP participantsconsider a greater proportion of the profile-relatedinformation compared with CBC respondents whenmaking their choices. This increased effort is alsoreflected in the time spent by participants on the task.The average time used by CP participants to com-plete 20 rounds of CP was 1,335.44 seconds, comparedto an average time of 625.76 seconds to complete 20CBC questions in the other condition. To the extentthat a longer response time is a sign of increasedinformation processing, this is not necessarily a nega-tive feature. However, from a managerial perspective,a longer response time may indicate that consumerswould demand higher incentives to participate in aCP study compared to a CBC study (holding the num-ber of rounds/questions constant). This would implythat for a fixed budget, more consumers could be sur-veyed using CBC compared to CP.
Our follow-up questionnaire allowed us to investi-gate these issues. Respondents were asked to indicateon a five-point scale from 1 (“strongly disagree”) to5 (“strongly agree”) how much they agreed with thefollowing statements:
• “Participation in this study was fun.”• “The instructions were complex.”• “Participation in this study required a lot of
effort.”• “Participation in this study required a lot of
time”In addition, in the spirit of Dong et al. (2010),
we measured how much money respondents in each
condition would require in order to participate in asimilar study in the future. Participants were giventhe following instructions:
We often run studies similar to this one, using thesame methodology but on products different from lap-top computers. Would you like to participate in suchfuture studies? Assuming that your only compensa-tion would be a fixed payment (i.e., you would notbe entered into a lottery to win a product), how muchwould that compensation need to be for you to bewilling to participate in a similar study of the samelength? Please enter an amount in dollars below. Wewill invite you to a future study of the same lengthonly if the payment for that study is at least as largeas the amount that you write. Otherwise, we will notsend you an invitation.
As expected, compared to participants in theincentive-compatible CBC condition, participants inthe CP condition found the instructions significantlymore complex (2.81 versus 1.74; Wald = 121003,p < 0001), found that the study required significantlymore effort (2.70 versus 1.84; Wald = 79028, p < 0001),and found that it was significantly more time con-suming (2.96 versus 2.06; Wald = 79056, p < 0001).However, they also thought that the study was sig-nificantly more “fun” (average scores of 4.12 ver-sus 3.97; Wald = 6090, p < 0001), and the amountsof money that they requested in order to participatein a similar survey in the future were similar. Inboth conditions, the median was equal to $5.00. (Theaverage was significantly higher in the CBC condi-tion because of the presence of outliers.) Moreover, aKolmogorov–Smirnoff (K-S) test could not reject thenull hypothesis that the amounts provided in the two
00082, p = 0023).To conclude this analysis, incentive-compatible CP
is perceived as being more complex, more effortful,and more time consuming, and it is objectively moretime consuming compared with incentive-compatibleCBC. But it is also perceived to be more enjoyable,and consumers do not request higher incentives toparticipate in a CP versus a CBC study, holding thenumber of rounds/questions constant.
Knowledge of Regular Poker. Our follow-up ques-tionnaire also tested the respondents’ knowledgeof traditional poker using a six-question quiz (seeAppendix D—none of the answers to these questionscould be inferred from the previous tasks). We cre-ated a score for each respondent equal to the num-ber of correct answers. First, as expected, we foundthat there was no significant difference in the aver-age score across the two conditions (4.48 versus 4.55;t = 00708, p-value = 0048). The proportion of respon-dents with perfect scores was also not significantlydifferent across the two conditions (36.79% versus34.59%; z = 00579, p = 0072). We tested whether CPaffects the partworths estimates differently for pokerexperts versus nonexperts. Although we may expectpoker experts to have preferences that systematicallydiffer from those of other consumers (related to dif-ferences in gender, age, etc.), it is important that theCP interface does not bias or distort the preferences ofpoker experts relative to nonexperts. We reestimatedthe partworths in each condition with a first-stageprior of �i ∼ N4�0 +�1Experti1D�5, where Experti wasa binary variable equal to 1 if respondent i receiveda perfect score on the poker quiz. The vector �1 wasestimated directly within the MCMC sampler (see, forexample, Lenk et al. 1996). The 95% credible inter-vals for all 24 attribute levels captured by �1 in theCBC condition overlap with those in the CP condi-tion. This suggests that the CP task itself does not dis-tort or biases the preferences of poker experts relativeto nonexperts.
Our results so far are consistent with CBC respon-dents making their choices based on a smaller subsetof the profile-related information presented to themcompared with CP respondents. The following twoeye-tracking studies provide direct evidence that CPparticipants consider more of the profile-related infor-mation presented to them compared with CBC par-ticipants. These studies also shed some light on thesource of this difference, by separating the effect ofdifferences in designs (i.e., sets of cards) from theeffect of differences in the methods themselves.
3.3. Study 2Eye-tracking has been used by several authors in mar-keting to model visual attention for brands, ads, and
in-store marketing (e.g., Chandon et al. 2009, Pietersand Wedel 2004, Wedel and Pieters 2000). In our set-ting, if CBC participants make their choices based ona smaller subset of attributes, they should consider asmaller proportion of the profile-related informationavailable to them, resulting in decreased visual atten-tion for a large proportion of attributes and levels.
Participants in our eye-tracking studies wererecruited at a large European university. The studywas run in a special eye-tracking lab, with a free-standing, nonintrusive eyetracker. The eye-trackingapparatus was a Tobii® 2150 tracker, samplinginfrared corneal reflections at 50 Hz with a 0.35� spa-tial resolution and an accuracy of 0.5�. Stimuli werepresented on the 21-inch LCD monitor of the eyetracker, controlled by a PC with a display resolutionof 1,600 × 1,200 pixels. The position of the partici-pant’s left eye and right eye was recorded separately(Van der Lans et al. 2011). After calibration of the eyetracker, respondents participated in the online studyon the eye-tracking screen in exactly the same man-ner as in the main experiment. Each participant com-pleted both CBC and CP in a counterbalanced order(in both Studies 2 and 3, similar results are obtainedwhen using data from the first task only, thereby cre-ating a between-subjects design).
Eye movements primarily consist of fixations andsaccades (Wedel and Pieters 2000). Fixations are dis-crete periods of time (about 200–500 milliseconds)where the eye hardly moves. During this phase,information is extracted. Saccades are quick jumps(20–40 milliseconds) between fixation locations. Toidentify eye-fixations from the recordings of the pointof regard, we used a recently developed velocity-based algorithm (Van der Lans et al. 2011). Each CBCquestion and each CP round involved four cards withsix attributes each. We identified the coordinates ofthe 6 × 4 = 24 cells on the screen that correspond toeach of these 24 areas, and we computed the numberof fixations in each of these 24 cells.
In this study, the respective designs (i.e., sets ofcards) used for CBC and CP were identical to thoseused in Study 1, and the sample size was N = 17. Theresults confirm that CP participants consider more ofthe available information compared with incentive-compatible CBC participants. In particular, CP par-ticipants considered (i.e., had at least one fixation inthe hand selection stage or card selection stage) onaverage 21.71 of the 24 cells containing the descrip-tion of the cards (where the average is taken acrossthe 20 rounds and across all participants), whereasincentive-compatible CBC participants considered onaverage 14.30 of the 24 cells.12 The difference in
12 In the hand selection stage of Conjoint Poker only, participantsconsidered on average 20.35 of the 24 cells, and in the card selection
Figure 7 Study 2—Average Proportion of Cells Considered by Participants in Each Condition
0 2 4 6 8 10 12 14 16 18 200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Round/question number
Pro
port
ion
of c
ells
vis
ited
Incentive-compatible CBCCP
Note. Each cell contains the level of one attribute for one alternative.
the average number of cells across respondents isstatistically significant (paired t-test, p-value < 0001),and all respondents considered more cells on aver-age under CP versus CBC. The eye-tracking data alsosuggest that the amount of information consideredby respondents tends to decrease as the study pro-gresses; see Figure 7. A binomial regression with alogistic link of the proportion of cells visited in eachquestion on the question number reveals a significantnegative trend for CBC respondents (p-value < 0001)and a marginally significant negative trend for CPrespondents (p-value ≈ 000850
We also report the amount of profile-related infor-mation considered in the external validity task. Witheight profiles, there are 6 × 8 = 48 cells. We computethe average number of cells considered by partici-pants in the external validity task the first time theywere exposed to the task (irrespective of whether theycompleted CBC or CP first; the external validity taskcame before the main task). Participants consideredon average 33.24 of the 48 cells. The fact that morethan 30% of the cells were not considered on aver-age raises questions on the ability of such externalvalidity tasks to capture real-world decisions. Futureresearch may explore whether our finding replicatesin other settings and may possibly develop alterna-tive measures to assess how well various preference
stage only, they considered on average 11.79 of the 18 cells on thescreen (in this stage only three cards remain on the screen). Notethat the proportion of cells considered in CBC should not be com-pared directly to the proportion of cells considered in the cardselection stage of Conjoint Poker, because it is likely that respon-dents remember at least part of the product-related informationacquired during the hand selection stage in the card selection stage.In Study 3, Conjoint Poker participants considered on average 20.72of the 24 cells in the hand selection stage and 11.77 of 18 in the cardselection stage.
measurement methods explain and predict real-worlddecisions.
3.4. Study 3Study 2 provided direct evidence that with the exper-imental design of Study 1, respondents consider moreof the profile-relevant information presented to themunder CP versus CBC. However, this finding mayalso be driven by differences in the designs (i.e.,set of cards) used in the two conditions. In particu-lar, the minimum-overlap D-efficient design used inthe CBC condition is such that it is very easy for arespondent to choose profiles based on one attributelevel (or a very small set of attribute levels). Study 3addresses this alternative explanation by holding theset of cards constant across the two conditions. In thisstudy, both conditions used the CP design used inStudies 1 and 2. Whereas this design is statisticallyefficient for CP, there is no apparent reason why itshould increase the amount of information consid-ered by respondents differently for one method ver-sus the other. Unfortunately, it was not possible to runa study in which both methods used the CBC designfrom Studies 1 and 2, because this design would leadto all hands in all rounds in CP having the samestrength (minimal overlap implies that all hands aredouble straights). Study 3 was otherwise identical toStudy 2, with N = 18.
The results confirm that CP participants considermore of the profile-related information comparedwith incentive-compatible CBC participants, evenholding the set of cards constant. In particular, CPparticipants considered (i.e., had at least one fixationin the hand selection stage or card selection stage) onaverage 21.63 of the 24 cells containing the descrip-tion of the cards (where the average is taken across
Figure 8 Study 3—Average Proportion of Cells Visited by Participants in Each Condition
0 2 4 6 8 10 12 14 16 18 200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Round/question number
Pro
port
ion
of c
ells
vis
ited
Incentive-compatible CBCCP
Note. Each cell contains the level of one attribute for one alternative.
the 20 rounds and across all participants), whereasincentive-compatible CBC participants considered onaverage 16.99 of the 24 cells. The difference in theaverage number of cells across respondents is statisti-cally significant (paired t-test, p-value < 0001), and 17of the 18 respondents considered more cells on aver-age under CP versus CBC. The downward trend inthe amount of information considered by respondentsas the questionnaire progresses is also confirmed andis now statistically significant both for CBC (p-value <0001) and CP (p-value < 0004); see Figure 8. The aver-age number of cells considered by respondents inthe external validity task was slightly lower than inStudy 2 and equal to 28.56 (out of 48).
In conclusion, Study 3 suggests that the differencesin the amount of information considered by respon-dents, which were observed indirectly in Study 1 anddirectly in Study 2, were at least partly driven byinherent differences between the methods and notonly by differences in designs (i.e., sets of cards)across conditions. The increase in the amount ofprofile-related information processed by CP partici-pants is at least partly a result of the structure of thetask; i.e., CP induces respondents to attend to profile-related information that they may otherwise ignore ina repeated choice task.
4. ConclusionsMarketing researchers have become increasinglyinterested in improving the quality of preferencemeasurement data. For instance, a stream of researchin the survey research literature deals with the identifi-cation of various effects that can influence the accuracyof people’s responses (see, for example, Baumgartnerand Steenkamp 2001, Tourangeau et al. 2000) and with
mechanisms that may improve the accuracy of theseresponses (see, for example, De Jong et al. 2010, whodeveloped a mechanism for the reporting of sensitiveinformation).
Similar concerns have been voiced in conjointanalysis research. One major recent contributionto the preference measurement literature has beenthe development of incentive-compatible mechanismsthat motivate respondents to reveal their prefer-ences truthfully. Here, we build on that literature topropose an incentive-compatible preference measure-ment mechanism that provides formal incentives inan engaging environment. Based on an online studyand two eye-tracking studies, we provide evidencethat this approach increases the amount of profile-related information considered by respondents.
Several areas for future research may be identi-fied. First and as noted above, future research mayexplore the generalizability and replicability of ourfinding that incentive-compatible CBC predicts out-of-sample individual choices well but predicts out-of-sample aggregate choice shares poorly. In our study,this result seems to have been linked to the exis-tence of groups of CBC respondents who all gave theexact same answers to all CBC questions, where theseanswers relied on a small subset of the attributes.13
13 Consider, for example, the group of 29 CBC respondents whoalways selected laptops with 320 GB hard drives in all calibra-tion questions. Both external validity profiles with 320 GB harddrives are attractive to this group of consumers. Because all of theserespondents gave the exact same set of answers to all calibrationquestions, the set of predicted out-of-sample choice probabilitiesis identical for all these respondents. Moreover, our ability to pre-dict which of these two 320 GB laptops will be chosen by theserespondents in the external validity task relies only on the use ofshrinkage. In other words, the out-of-sample predictions within the
Second, additional eye-tracking studies may be runto explore whether alternative conjoint analysis for-mats, such as rating-based conjoint analysis, are ableto increase information consideration. Third, futureresearch may identify situations in which deeperinformation processing in preference measurementtasks is actually not desirable, e.g., because it inducesrespondents to pay more attention to information thatthey would ignore when making real-life decisions.A fourth area for future research is the developmentand testing of multiplayer versions of Conjoint Poker.This would raise several issues, including (a) play-ers may be able to learn each other’s preferences;(b) players may be able to learn each other’s play-ing style and, in particular, the propensity to playthe strongest hand; (c) the assumptions made inAppendix A that the opponent’s cards are drawnrandomly and that the opponent always plays thestrongest available hand would not hold anymore,and the computation of the probabilities of winningwith each hand would be more complicated; and(d) the efficiency of each player’s design would beinfluenced by each other player’s design (throughthe probabilities of winning), and therefore designsshould be optimized jointly across players.
Electronic CompanionAn electronic companion to this paper is available as part ofthe online version that can be found at http://mktsci.journal.informs.org/.
AcknowledgmentsThe authors thank Ralf van der Lans for his help with theanalysis of eye-tracking data. M. G. de Jong thanks the NielsStensen Foundation and the Netherlands Organization forScientific Research for financial support.
Appendix A. Hand Probabilities
A.1. Hand StrengthsWe show here how to compute the probabilities that eachhand is achievable from a random set of four cards drawnuniformly without replacement from the set of all possiblecards in the case of A = 6 attributes with four levels each.The same approach may be used for any other number ofattributes and levels. These probabilities capture the proba-bility that each hand is achievable by the computer in ourstudies (against which our respondents played) and will beused next to compute the probability of winning with eachhand when playing against the computer.
small set of profiles with 320 GB are “random guesses” driven onlyby population-level shrinkage. Because there are only two profileswith 320 GB, hit rates among these respondents are high (∼50%).However, aggregate choice share predictions among these respon-dents are less accurate if the relative shares of these two profilesare not well predicted.
Let b equal to the probability that four random cards haveat least three different levels on a given attribute: b = 4
(42
)
·
4 · 3 · 25/44 + 4!/44 = 42/64.Let c be the probability that at least three of four random
cards have the same level on a given attribute: c = 44 ·4 ·35/44 + 4/44 = 13/64.
Let d = 1−b−c be the probability that four random cardsare such that two have one level and two have another levelon a given attribute: d = 44 · 3 · 35/44 = 9/64.
Let e the probability that exactly three of four randomcards have the same level on a given attribute given thatthey have at least three: e = 444 · 4 · 35/445/413/645= 12/13
Let f be the probability that exactly three of four randomcards have different levels on a given attribute given that atleast three do: f = 44
(42
)
· 4 · 3 · 25/445/442/645= 36/42.We have the following:
Pr(Double Flush)
=
A∑
k2=2
(
A
k2
)
41 − c5A−k2ck2
(
14k2 > 45+ 14k2 ≤ 45
·
(
1 − ek23!
44 − k25!
(
14
)k2−1))
0
Pr(Straight Flush)
=
A−1∑
k2=1
A−k2∑
k1=1
(
A
k1
)
·
(
A− k1
k2
)
dA−k1−k2bk1ck2
·
[
1 − f k1ek2
((
12
)k1(14
)k2−1
+
(
42
)(
12
)k2
·
(
1 −
(
12
)k2−1)(16
)k1)]
0
Pr(Flush) = 1 − 41 − c5A0
Pr(Double Straight)
=
A∑
k1=2
(
A
k1
)
· 41 − b5A−k1bk1
·
[
14k1 ≥ 35+ 14k1 ≤ 25·(
1 − f k1
(
16
)k1−1)]
0
Pr(Straight) = 1 − 41 − b5A0
Pr(Pair) = 1 − 4b41 − f 55A0
A.2. Probabilities of Winning Against the ComputerFor each round and for each possible hand, we developclosed-form expressions for the probability of winning ifplaying that hand against a player with random cards whoalways plays the best hand. We compute Pr(best hand isDouble Flush), Pr(best hand is Straight Flush), Pr(best handis Flush), Pr(best hand is Double Straight), Pr(best handis Straight), and Pr(best hand is Pair). (It is easy to showthat Pr(not getting anything) = 0.) Then the probability of
winning with a given hand is the sum of probabilities thatthe opponent will get a weaker hand plus half the proba-bility that the opponent will get a hand of similar strength.
We have the following:
Pr(best hand is Double Flush)
=
A∑
k2=2
(
A
k2
)
41 − c5A−k2ck2
(
14k2 > 45+ 14k2 ≤ 45
·
(
1 − ek23!
44 − k25!
(
14
)k2−1))
0
Pr(best hand is Straight Flush)
= Pr(Straight Flush, no Double Flush)
=
4∑
k2=1
A−k2∑
k1=1
(
A
k1
)(
A− k1
k2
)
dA−k1−k2bk1ck2
·
[
14k2 = 15+ 14k2 > 15ek23!
44 − k25!
(
14
)k2−1]
·
14k2 > 25+ 14k2 ≤ 25
1 − f k1
(
(4−k22
)
6
)k1
0
Pr(best hand is Flush)
= Pr(Flush, no Straight Flush, no Double Flush)
=
4∑
k2=1
A−k2∑
k1=0
(
A
k1
)(
A− k1
k2
)
dA−k1−k2bk1ck2
·
[
14k2 = 15+ 14k2 > 15ek23!
44 − k25!
(
14
)k2−1]
·
14k1 = 05+ 14k1 > 05 · 14k2 ≤ 25 · f k1
(
(4−k22
)
6
)k1
0
Pr(best hand is Double Straight)
= Pr(Double Straight, no Flush,
no Straight Flush, no Double Flush)
=
A∑
k1=2
(
A
k1
)
dA−k1bk1
[
14k1 ≥ 35+ 14k1 = 25 ·56
]
0
Pr(best hand is Straight)
= Pr(Straight, no Double Straight, no Flush,
no Straight Flush, no Double Flush)
=
2∑
k1=1
(
6k1
)
d6−k1bk1
[
14k1 = 15+ 14k1 = 25 ·16
]
0
Pr(best hand is Pair)
= Pr(Pair, no Straight, no Double Straight,
no Flush, no Straight Flush, no Double Flush)
= dA0
Appendix B. Computation of theInformation Matrix
B.1. Homogeneous CaseThe log of the likelihood corresponding to a given round rin which the player played h and chose profile j is
Lr = �V rh − log
(
∑
h′
exp4�V rh′ 5
)
+ xj�− log(
∑
j ′∈h
exp4xj ′�5)
1
ïLr = �ïV rh −
∑
h′ exp4�V rh′ 5�ïV r
h′
∑
h′ exp4�V rh′ 5
+ xTj −
∑
j ′∈h exp4xj ′�5xTj ′∑
j ′∈h exp4xj ′�5
= �ïV rh −
∑
h′
Pr(h’)�ïV rh′ + xTj −
∑
j ′∈h
Pr4j ′5xTj ′ 0
The information matrix for round r is given by
ìr = Ehr 1j44ïLr 54ïLr 5
T 5
=∑
hr
∑
j∈hr
[
�ïV rhr
−∑
h′r
Pr4h′
r 5�ïVrh′r+xTj −
∑
j ′∈hr
Pr4j ′5xTj ′]
·
[
�ïV rhr
−∑
h′r
Pr4h′
r 5·�ïVrh′r+xTj −
∑
j ′∈hr
Pr4j ′5xTj ′]T
·Pr4hr 5Pr4j �hr 5
=∑
hr
[
�ïV rhr
−∑
h′r
Pr4h′
r 5�ïVrh′r
]
·Pr4hr 5
[
�ïV rhr
−∑
h′r
Pr4h′
r 5�ïVrh′r
]T∑
j∈hr
Pr4j �hr 5
+∑
hr
Pr4hr 5∑
j∈hr
[
xTj −∑
j ′∈hr
Pr4j ′ �hr 5xTj ′
]
·Pr4j �hr 5
[
xTj −∑
j ′∈hr
Pr4j ′ �hr 5xTj ′
]T
+∑
hr
[
�ïV rhr
−∑
h′r
Pr4h′
r 5�ïVrh′r
]
·Pr4hr 5∑
j∈hr
[
xjT−∑
j ′∈hr
Pr4j ′ �hr 5x′
jT
]T
Pr4j �hr 5
=∑
hr∈r
4z̃rhr Pr4hr 5z̃Thrr
+Pr4hr 5∑
j∈hr
zj �hr Pr4j �hr 5zTj �hr
51
where
z̃rhr = �ïV rhr
−∑
h′r
�ïV rh′r
Pr4h′
r 5 and
zj �hr = xjT
−∑
j ′∈hr
x′
jT Pr4j ′ � hr 50
B.2. Mixed Logit CaseIn the case of a mixed logit model, following Sandor andWedel (2002), we assume that �i = � + Ui� , where � is adiagonal matrix and Ui = diag4ui5, where ui ∼ N401 I5. Let
The log of the likelihood corresponding to a givenround r in which the player chose hand h and profile j maybe written as
Lr = log4�h5+ log4�j �h51
¡Lr
¡�=
1�h
¡�h
¡�+
1�j �h
¡�j �h
¡�
=1�h
∫
u
¡Pr4h � u5
¡�ê4u5du
+1
�j �h
∫
u
¡Pr4j � h1u5
¡�ê4u5du
=1�h
∫
u
[
4�ïVhr Pr4h � u5
−∑
h′
Pr4h′� u5Pr4h � u5�ïV ′
hr
]
ê4u5du
+1
�j �h
∫
u
U
[
4xjT Pr4j � h1u5
−∑
j ′∈h
Pr4j � h1u5Pr4j ′ � h1u5xTj ′
]
ê4u5du1
¡Lr
¡�=
1�h
¡�h
¡�+
1�j �h
¡�j �h
¡�
=1�h
∫
u
¡Pr4h � u5
¡�ê4u5du
+1
�j �h
∫
u
¡Pr4j � h1u5
¡�ê4u5du
=1�h
∫
uU
[
4�ïVhr Pr4h � u5
−∑
h′
Pr4h′� u5Pr4h � u5�ïV r
h′
]
ê4u5du
+1
�j �h
∫
uU
[
4xjT Pr4j � h1u5−
∑
j ′∈h
Pr4j � h1u5
· Pr4j ′ � h1u5x′
jT
]
ê4u5du1
ìr =
∣
∣
∣
∣
∣
∣
∣
∣
∣
Ehr1 j
((
¡Lr
¡�
)(
¡Lr
¡�
)T)
Ehr1 j
((
¡Lr
¡�
)(
¡Lr
¡�5T)
1
Ehr 1 j
((
¡Lr
¡�
)(
¡Lr
¡�
)T)
Ehr 1 j
((
¡Lr
¡�
)(
¡Lr
¡�
)T)
∣
∣
∣
∣
∣
∣
∣
∣
∣
1
where
Ehr 1 j
((
¡Lr
¡�
)(
¡Lr
¡�
)T)
=∑
hr
∑
j∈hr
(
¡Lr
¡�
)(
¡Lr
¡�
)T
�hr�j �hr
1
Ehr 1 j
((
¡Lr
¡�
)(
¡Lr
¡�
)T)
=∑
hr
∑
j∈hr
(
¡Lr
¡�
)(
¡Lr
¡�
)T
�hr�j �hr
1
Ehr 1 j
((
¡Lr
¡�
)(
¡Lr
¡�
)T)
=∑
hr
∑
j∈hr
(
¡Lr
¡�
)(
¡Lr
¡�
)T
�hr�j �hr
0
Appendix C. Hierarchical Bayes EstimationThe likelihood function for Conjoint Poker is given by Equa-tions (2) and (4). The likelihood function for CBC is givensimply by
Pr4j5=exp4xj�i5
∑
j ′∈r exp4xj ′�i50
Let p be equal to the number of partworths estimated(p = 18 in our case), and let Ip denote the identity matrixof size p. The same prior specification was used in bothconditions:14
�i ∼ N4�01D�51
D−1� ∼ Wishart40001Ip1 p+ 1551
diffuse improper prior on �0.
In addition, we used the following priors on the addi-tional parameters in Conjoint Poker:
�i ∼ TN4�01D� 5 4truncated between 0 and 251
�i ∼ N4�01D�51
D−1� ∼ Gamma411151
D−1� ∼ Gamma411151
diffuse improper prior for �0 and �01
diffuse improper prior on <+ for �0
We used MCMC with 50,000 iterations, using the first10,000 as burn-in and saving 1 in every 10 iterationsthereafter. Convergence was assessed visually using plotsof the parameters. We used rejection sampling on �i toensure that �i + maxj∈h8xj�i9≥ 0 for all i and h.
Appendix D. Poker QuizPlease answer the following questions regarding real poker.
The pot is—a set of cards placed in the middle of the table.—an amount of money for which players compete.—an amount of money that players may borrow during
the game.To call is
—to discard one’s hand and forfeit interest in the cur-rent pot.
—to increase the size of the bet required to stay in thepot.
—to match a bet or match a raise.To fold is
—to discard one’s hand and forfeit interest in the cur-rent pot.
—to increase the size of the bet required to stay in thepot.
—to match a bet or match a raise.An ante is
—a forced bet in which all players put an equal amountof money or chips into the pot before the deal begins.
14 Of course, each condition was estimated separately andindependently.
—the final community card.—the first three community cards.
A flop is—a forced bet in which all players put an equal amount
of money or chips into the pot before the deal begins.—the final community card.—the first three community cards.
A river is—a forced bet in which all players put an equal amount
of money or chips into the pot before the deal begins.—the final community card.—the first three community cards.
ReferencesBaumgartner, H., J.-B. E. M. Steenkamp. 2001. Response styles in
marketing research: A cross-national investigation. J. MarketingRes. 38(2) 143–156.
Bradlow, E. T. 2005. Current issues and a wishlist for conjoint anal-ysis. Appl. Stochastic Models Bus. Indust. 21(4–5) 319–323.
Bunch, D. S., J. J. Louviere, D. Anderson. 1994. A comparison ofexperimental design strategies for multinomial logit models:The case of generic attributes. Working paper, Graduate Schoolof Management, University of California, Davis, Davis.
Chandon, P., J. W. Hutchinson, E. T. Bradlow, S. H. Young. 2009.Does in-store marketing work? Effects of the number and posi-tion of shelf facings on brand attention and evaluation at thepoint of purchase. J. Marketing 73(November) 1–17.
De Jong, M. G., R. Pieters, J.-P. Fox. 2010. Reducing social desir-ability bias via item randomized response: An application tomeasure underreported desires. J. Marketing Res. 47(February)14–27.
Ding, M. 2007. An incentive-aligned mechanism for conjoint anal-ysis. J. Marketing Res. 44(1) 214–223.
Ding, M., R. Grewal, J. Liechty. 2005. Incentive-aligned conjointanalysis J. Marketing Res. 42(1) 67–82.
Ding, M., Y.-H. Park, E. T. Bradlow. 2009. Barter markets for con-joint analysis Management Sci. 55(6) 1003–1017.
Dong, S., M. Ding, J. Huber. 2010. A simple mechanism to incen-tive align conjoint experiments. Internat. J. Res. Marketing 27(1)25–32.
Green, P. E., V. Srinivasan. 1990. Conjoint analysis in marketing:New developments with implications for research and practice.J. Marketing 54(4) 3–19.
Huber, J., K. Zwerina. 1996. The importance of utility balance inefficient choice designs. J. Marketing Res. 33(3) 307–317.
Johnson, R. 2008. A perspective on adaptive CBC (what can weexpect from respondents?). Research paper, Sawtooth Software,Sequim, WA. http://www.sawtoothsoftware.com/download/techpap/acbcperspective.pdf.
Kohli, R., R. Sukumar. 1990. Heuristics for product-line designusing conjoint analysis. Management Sci. 36(12) 1464–1478.
Lenk, P. J., W. S. DeSarbo, P. E. Green, M. R. Young. 1996. Hier-archical bayes conjoint analysis: Recovery of partworth het-erogeneity from reduced experimental designs. Marketing Sci.15(1) 173–191.
Liechty, J. C., D. K. H. Fong, W. DeSarbo. 2005. Dynamic modelswith individual level heterogeneity: Applied to evolution dur-ing conjoint studies. Marketing Sci. 24(1) 285–293.
Mahajan, V., P. E. Green, S. M. Goldberg. 1982. A conjoint model formeasuring self- and cross-price/demand relationships. J. Mar-keting Res. 19(3) 334–342.
McFadden, D. 1974. Conditional logit analysis of qualitative choicebehavior. P. Zarembka, ed. Frontiers in Econometrics. AcademicPress, New York, 105–142.
Netzer, O., O. Toubia, E. T. Bradlow, E. Dahan, T. Evgeniou, F. M.Feinberg, E. M. Feit et al. 2008. Beyond conjoint analysis:Advances in preference measurement. Marketing Lett. 19(3–4)337–354.
Newey, W. K., D. McFadden. 1994. Large sample estimation andhypothesis testing. R. F. Engle, D. L. McFadden, eds. Handbookof Econometrics, Vol. 4. Elsevier Science, Amsterdam, 2113–2247.
Park, Y.-H., M. Ding, V. R Rao. 2008. Eliciting preference for com-plex products: A Web-based upgrading method. J. MarketingRes. 45(5) 562–574.
Pieters, R., M. Wedel. 2004. Attention capture and transfer inadvertising: Brand, pictorial, and text-size effects. J. Marketing68(April) 36–50.
Rossi, P. E., G. M. Allenby. 2003. Bayesian statistics and marketing.Marketing Sci. 22(3) 304–328.
Sandor, Z., M. Wedel. 2001. Designing conjoint choice experimentsusing managers’ prior beliefs. J. Marketing Res. 38(4) 430–444.
Sandor, Z., M. Wedel. 2002. Profile construction in experimentalchoice designs for mixed logit models. Marketing Sci. 21(4)455–475.
Starmer, C., R. Sugden. 1991. Does the random-lottery incentivesystem elicit true preferences? An experimental investigation.Amer. Econom. Rev. 81(4) 971–978.
Thaler, R. H., E. J. Johnson. 1990. Gambling with the house moneyand trying to break even: The effects of prior outcomes on riskychoices. Management Sci. 36(6) 643–660.
Tourangeau, R., L. J. Rips, K. Rasinski. 2000. The Psychology of SurveyResponse. Cambridge University Press, Cambridge, UK.
Van der Lans, R., M. Wedel, R. Pieters. 2011. Defining eye-fixation sequences across individuals and tasks: The binocular-individual threshold (BIT) algorithm. Behav. Res. Methods 43(1)239–257.
Wedel, M., R. Pieters. 2000. Eye fixations on advertisements andmemory for brands: A model and findings. Marketing Sci. 19(4)297–312.
Wittink, D. R., P. Cattin. 1989. Commercial use of conjoint analysis:An update. J. Marketing Res. 53(3) 91–96.
![Preference Elicitation [Conjoint Analysis]. Conjoint Analysis Market research: assess consumer’s preferences on homogenous class of products Approach:](https://static.documents.pub/doc/80x56/56649d1f5503460f949f25d8/preference-elicitation-conjoint-analysis-conjoint-analysis-market-research.jpg)