Expert Systems With Applications 61 (2016) 246–261

Contents lists available at ScienceDirect

Expert Systems With Applications

journal homepage: www.elsevier.com/locate/eswa

Reducing preference elicitation in group decision making

Lihi Naamani-Dery

a , ∗, Meir Kalech

b , Lior Rokach

b , Bracha Shapira

b

a Industrial Engineering and Management, Ariel University, Ariel 40700, Israel b Information Systems Engineering, Ben-Gurion University of the Negev, P.O.Box 653 Beer-Sheva 8410501, Israel

a r t i c l e i n f o

Article history:

Received 14 December 2015

Revised 9 May 2016

Accepted 27 May 2016

Available online 30 May 2016

Keywords:

Preference elicitation

Group decision making

Computational social choice

a b s t r a c t

Groups may need assistance in reaching a joint decision. Elections can reveal the winning item, but this

means the group members need to vote on, or at least consider all available items. Our challenge is to

minimize the amount of preferences that need to be elicited and thus reduce the effort required from

the group members. We present a model that offers a few innovations. First, rather than offering a single

winner, we propose to offer the group the best top- k alternatives. This can be beneficial if a certain item

suddenly becomes unavailable, or if the group wishes to choose manually from a few selected items.

Secondly, rather than offering a definite winning item, we suggest to approximate the item or the top- k

items that best suit the group, according to a predefined confidence level. We study the tradeoff between

the accuracy of the proposed winner item and the amount of preference elicitation required. Lastly, we

offer to consider different preference aggregation strategies. These strategies differ in their emphasis: to-

wards the individual users (Least Misery Strategy) or towards the majority of the group (Majority Based

Strategy) . We evaluate our findings on data collected in a user study as well as on real world and sim-

ulated datasets and show that selecting the suitable aggregation strategy and relaxing the termination

condition can reduce communication cost up to 90%. Furthermore, the commonly used Majority strategy

does not always outperform the Least Misery strategy. Addressing these three challenges contributes to

the minimization of preference elicitation in expert systems.

© 2016 Elsevier Ltd. All rights reserved.

i

U

i

P

a

&

N

d

p

e

p

L

o

f

p

v

c

a

1. Introduction

A group of people wishing to reach a joint decision faces the

task of selecting the alternative that best suits the group out of all

available candidate items. When all users’ preferences are known,

some voting aggregation strategy is used to compute and output

the winning item to the group ( Rossi, Venable, & Walsh, 2011 ).

When the preferences are not available, a preference elicitation

process is required.

Preference elicitation requires time and effort, so our goal is

to stop the elicitation as soon as possible. In the worst case, for

most voting protocols all the preferences are needed in order to

determine a winning item, i.e., an item that most certainly suits

the group’s joint preferences ( Conitzer & Sandholm, 2005 ). Nev-

ertheless, in practice it has been shown that the required infor-

mation can be cut in more than 50% ( Kalech, Kraus, Kaminka, &

Goldman, 2011; Lu & Boutilier, 2011 ). Given partial preferences, it

is possible to define the set of the necessary winners, i.e., items

which must necessarily win, as well as the set of possible winners,

∗ Corresponding author.

E-mail addresses: lihid@ariel.ac.il , nlihin@gmail.com (L. Naamani-Dery),

kalech@bgu.ac.il (M. Kalech), liorrk@bgu.ac.il (L. Rokach), bshapira@bgu.ac.il (B.

Shapira).

a

o

&

t

i

http://dx.doi.org/10.1016/j.eswa.2016.05.041

0957-4174/© 2016 Elsevier Ltd. All rights reserved.

.e., items which can still possibly win ( Konczak & Lang, 2005 ).

sing these definitions the elicitor can determine whether there

s need for more information concerning the voters’ preferences.

revious studies provide algorithms for preference elicitation of

single winner under the Range and the Borda protocols ( Lu

Boutilier, 2011; Naamani-Dery, Golan, Kalech, & Rokach, 2015;

aamani-Dery, Kalceh, Rokach, & Shapira, 2014 ). In this paper we

efine two tradeoffs that enable less elicitation: Selection and Ap-

roximation . Furthermore, we propose to examine different prefer-

nce Aggregation techniques.

Selection: a tradeoff exists between the amount of items out-

utted to the group and the cost of preferences elicitation required.

ess elicitation effort is required for outputting k items where one

f them is the winner with a high probability (top- k items) than

or outputting one necessary winner (i.e., k = 1 ). Although out-

utting a definite winner is the most accurate result, there are ad-

antages to outputting the top- k items. Not only is the communi-

ation cost reduced, it may actually be preferred to present a few

lternatives to the user since if one of the alternatives is unavail-

ble the group members can quickly switch to another already rec-

mmended alternative without requiring more elicitation ( Baldiga

Green, 2013; Lu & Boutilier, 2010 ). Consider, for example, a set-

ing of 30 optional dinner locations for a group. If a fish restaurant

s the winning item, but one of the group members dislikes fish,

L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261 247

t

t

t

o

t

r

w

p

o

t

a

T

a

p

t

t

t

A

1

g

u

c

i

g

m

e

s

(

o

e

c

T

b

fi

s

p

a

W

n

I

q

c

s

(

w

a

i

a

t

(

i

c

t

s

c

w

r

a

d

b

a

u

(

p

(

o

2

a

o

g

g

o

t

2

i

p

o

2

M

i

w

s

g

c

i

t

o

s

T

a

f

i

a

w

t

t

M

i

s

r

a

2

t

s

v

a

i

he group might prefer to switch to a different alternative rather

han to perform another elicitation round.

Approximation: a different tradeoff is the one that exists be-

ween the accuracy of the proposed winner item and the amount

f preference elicitation required. We suggest outputting an item

hat approximately suits the group with some confidence level

ather than outputting an item that definitely suits the group. As

e later show, the confidence level is based on the items’ winning

robabilities. To reduce the elicitation even further, the two meth-

ds can be combined and top- k approximate items can be offered

o the group. Consider, for example, a group that wishes to choose

movie to watch together out of movies available in the cinema.

he members set the amount of options they wish to receive ( k )

nd the level of confidence of the results. Thus, we define a new

reference elicitation termination condition: approximate k-winner

ermination , namely where k items are found and one of them is

he best item with a confidence level of 1 − α (0 ≤ α ≤ 1).

Aggregation: Ideally, the preference aggregation strategy (i.e.,

he voting protocol) should be a fair one. In his well-known work,

rrow shows that there is no perfect aggregation system ( Arrow,

951 ). One of the major differences between aggregation strate-

ies is the social environment in which they are used; in partic-

lar, the perspective in which fairness is viewed. The emphasis

an be either towards the individual user or towards the major-

ty of the group ( Jameson & Smyth, 2007 ). Two aggregation strate-

ies that differ in their emphasis and are used in group recom-

ender systems are the Majority Based Strategy and the Least Mis-

ry Strategy ( Masthoff, 2011 ). Similar concepts can be found in the

ocial choice literature, termed utilitarianism and egalitarianism

Myerson, 1981 ). In the Majority Based Strategy the users’ ratings

f the different items are aggregated and the items with the high-

st total value are recommended. In the Least Misery Strategy the

hosen items cannot be the least preferred by any of the users.

he idea is that a group is as happy as its least happy mem-

er. One of the contributions of this paper is in proposing an ef-

cient iterative preference elicitation algorithm which fits these

trategies.

Overall, our goal is to reduce the communication cost in the

reference elicitation process. We define the communication cost

s the cost of querying one user for her preferences for one item.

e allow users to submit the same rating for many items and do

ot request the users to hold a strict set of preferences over items.

n this paper, we adopt the Range voting protocol which is ade-

uate for this purpose; it requires users to submit a score within a

ertain range. Users are familiar with applications that ask for their

core on an item, such as Amazon ( www.amazon.com ) or Netflix

www.netflix.com ).

Preference elicitation becomes more challenging and interesting

hen a rating distribution of the voter-item preferences exists, i.e.,

prior probability distribution of each voter’s preferences for each

tem. For example, in the case of a group of users wish to watch

movie together, the distribution can be inferred from rankings of

hese movies by similar users using collaborative filtering methods

Koren & Sill, 2011 ). After each user-item query, new information

s revealed. The necessary and possible winner sets are updated to

heck whether or not the termination condition has been reached.

In this paper, we offer three main innovations contributing to

he minimization of preference elicitation:

1. Selection : we suggest terminating preference elicitation

sooner by returning k alternatives to the group members

rather than returning just one item.

2. Approximate winners : we suggest computing approximate

winner or winners with some confidence level. This as well

reduces the communication cost.

3. Aggregation : we suggest considering the Least Misery aggre-

gation Strategy beyond the known Majority based strategy

We evaluated the approach on multiple datasets in different

cenarios and application domains: (1) Two datasets that were

ollected using a group recommender system named “Lets Do It”

hich was built and operated in Ben-Gurion University. (2) Two

eal world datasets, the Netflix data ( http://www.netflixprize.com )

nd Sushi data ( Kamishima, Kazawa, & Akaho, 2005 ). (3) Simulated

ata which allow us to study the impact of the probability distri-

ution. We show that selecting the suitable aggregation strategy

nd relaxing the termination condition can reduce communication

p to 90%.

This paper is an extension of the authors’ previous short paper

Naamani-Dery, Kalech, Rokach, & Shapira, 2014 ). In the previous

aper, we shortly presented one preference elicitation algorithm

DIG) without approximation. In this paper we added an overview

f the state of the art in the field of voting techniques ( Section

). We extended the model and definitions and added a model for

pproximation of the necessary winner ( Section 3 ). We added an-

ther algorithm, ES ( Section 4 ). This allows us to compare the al-

orithms’ performance in different settings and show that each al-

orithm has an advantage in different scenarios. We have extended

ur evaluation to include a user-study, detailed experiments and a

horough analysis ( Sections 5 and 6 ).

. Related work

Group decision making consists of two phases: preference elic-

tation and preference aggregation. We start with describing the

reference elicitation strategies considered in this paper, and move

n to describe how preference elicitation.

.1. Preference aggregation strategies

One of the contributions of this paper is to consider the Least

isery strategy, which, to our best knowledge, has not been stud-

ed in the context of preference elicitation . Throughout this paper

e use “Majority” and “Least Misery” to refer to the Majority based

trategy and the Least Misery based strategy ( Masthoff, 2004 ).

Different studies have shown how different strategies affect

roup members ( Masthoff, 2004; Senot, Kostadinov, Bouzid, Pi-

ault, & Aghasaryan, 2011 ). Masthoff studies how humans prefer to

ntegrate personal recommendations. She concludes that users use

he Majority Strategy, the Least Misery strategy and Majority with-

ut Misery strategy ( Masthoff, 2004 ). Her findings motivate our re-

earch to focus on the Majority and the Least Misery strategies.

hese two strategies were also chosen by Baltrunas, Makcinskas,

nd Ricci (2010) , in research focusing on the evaluation of the ef-

ectiveness of Group Recommender Systems obtained by aggregat-

ng user preferences.

In the Majority Strategy the users’ ratings of the different items

re aggregated and the item with the highest total value is the

inner. Note that the result is similar to selecting the item with

he highest average, thus this strategy is sometimes referred to as

he Average Strategy or the Additive Strategy ( Masthoff, 2011 ). The

ajority strategy is used in numerous applications. For example:

n the MusicFX system the square of the individual preferences are

ummed ( McCarthy & Anagnost, 1998 ) and the Travel Decision Fo-

um assists in planning a holiday ( Jameson, 2004 ). Yet another ex-

mple is of TV programs recommendation for a group ( Masthoff,

004; Yu, Zhou, Hao, & Gu, 2006 ), where the chosen program fits

he wishes of the majority of the group. A disadvantage of this

trategy is that it can be unfair towards users with the minority

iew. In fact, Yu et al. (2006 ) state that their system works well for

homogenous group but when the group is heterogeneous, dissat-

sfaction of the minority group occurs.

248 L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261

W

c

v

a

fl

i

m

d

2

f

c

t

a

t

K

d

e

i

a

t

t

u

n

l

t

c

t

h

e

t

f

p

q

a

e

e

i

g

F

m

p

s

a

3

w

u

d

a

a

t

3

a

f

c

i

D

R

The Least Misery Strategy defines that the chosen items cannot

be the least preferred by any of the users. In the Polylens system

the Least Misery strategy is used to recommend movies to small

groups ( O’connor, Cosley, Konstan, & Riedl, 2002 ). Survey results

show that 77% of the users found the group recommendation more

helpful than the personal one. The disadvantage is that the minor-

ity opinion can dictate the decision for the entire group – if all

the users of the group except one prefer some item, it is still not

chosen ( Masthoff, 2011 ).

2.2. Preference elicitation

Voting with partial information, i.e., when voters do not re-

veal their preferences for all candidates, has a theoretical basis

( Conitzer & Sandholm, 2005; Konczak & Lang, 2005 ). Conitzer and

Sandholm (2005) analyze the communication complexity of vari-

ous voting protocols and determine upper and lower bounds for

communication costs (communication with the users). In general,

they show that for most voting protocols, in the worst possible

case voters should send their entire set of preferences. Konczak

and Lang (2005) demonstrate how to compute the set of possi-

ble winners and a set of necessary winners. These sets determine

which candidates no longer have a chance of winning and which

will certainly win. We adopt their approach and propose a system

where the agents do not need to send their entire set of prefer-

ences.

A theoretical bound for the computation of necessary winners

has been addressed ( Betzler, Hemmann, & Niedermeier, 2009; Pini,

Rossi, Venable, & Walsh, 2009; Walsh, 2007 ) and so has the the-

oretical complexity of approximating a winner ( Service & Adams,

2012 ). The complexity of finding possible winners has also been in-

vestigated ( Xia & Conitzer, 2008 ). Others considered settings where

preferences may be unspecified, focusing on soft constraint prob-

lems ( Gelain, Pini, Rossi, & Venable, 2007 ) or on sequential ma-

jority voting ( Lang, Pini, Rossi, Venable, & Walsh, 2007 ). They do

not provide empirical evaluation nor do they focus on reducing

the communication load. Ding and Lin (2013) define a candidate

winning set as the set of queries needed in order to determine

whether the candidate is a necessary winner. The authors show

that for rules other than Plurality voting, computing this set is NP-

Hard. Following this theorem, we propose heuristics for preference

elicitation.

Prior probability distribution of the votes is assumed by Hazon,

Aumann, Kraus, and Wooldridge (2008) . The winning probability

of each candidate is evaluated using Plurality, Borda and Copeland

protocols ( Brams & Fishburn, 2002 ). Hazon et al. also show the-

oretical bounds for the ability to calculate the probability of an

outcome. Bachrach, Betzler, and Faliszewski (2010) provide an al-

gorithm for computing the probability of a candidate to win, as-

suming a voting rule that is computable in polynomial time (such

as range voting) and assuming a uniform random distribution of

voters’ choice of candidates. However, while both Hazon et al. and

Bachrach et al. focus on calculating the winning probability for

each candidate, we focus on practical vote elicitation and specifi-

cally on approximating the winner within top- k items using a min-

imal amount of queries.

Practical vote elicitation has been recently addressed using var-

ious approaches. Pfeiffer, Gao, Mao, Chen, and Rand (2012) predict

the ranking of n items; they elicit preferences by querying voters

using pairwise comparison of items. However, they do not explic-

itly aim to reduce the number of queries. Furthermore, they as-

sume that each voter can be approached only once and that there

is no prior knowledge about the voters. As a result, voter-item dis-

tributions cannot be computed. Their method is therefore suitable

when a large amount of voters is available and the task is to deter-

mine some hidden truth (also known as the wisdom of the crowd).

e, on the other hand, wish to reach a joint decision for a spe-

ific group of voters. Chen and Cheng (2010) allow users to pro-

ide partial preferences at multiple times. The authors developed

n algorithm for finding a maximum consensus and resolving con-

icts. However, they do not focus on reducing the preference elic-

tation required. Another approach is to require the voters to sub-

it only their top- k ranked preferences and these are used to pre-

ict the winning candidate with some probability ( Filmus & Oren,

014; Oren, Filmus, & Boutilier, 2013 ). This usage of top- k is dif-

erent than ours; we ask users to submit their rating for certain

hosen items, and output top- k alternatives to the group. All of

hese methods address vote elicitation, however they do not aim

t reducing the preference elicitation process, i.e., they do not try

o minimize the amount of queries the users receive.

An attempt to reduce the number of queries is presented by

alech et al. (2011) . They assume that each user holds a predefined

ecreasing order of the preferences. In an iterative process, the vot-

rs are requested to submit their highest preferences; the request

s for the rating of a single item from all the users. One major dis-

dvantage of this approach is that requiring the users to predefine

heir preferences can be inconvenient to the users. Another prac-

ical elicitation process is proposed for the Borda voting protocol

sing the minmax regret concept. The output is a definite win-

er or an approximate winner, but the approximation confidence

evel is not stated ( Lu & Boutilier, 2011 ). The method was later ex-

ended to return multiple winners, again using the Borda proto-

ol and minmax regret ( Lu & Boutilier, 2013 ). Two practical elicita-

ions algorithms that aim to minimize preference communication

ave been presented for the Range voting protocol ( Naamani-Dery

t al., 2014 ). In this paper, we use these algorithms as the basis

o address the presented challenges. We iteratively query one voter

or her preferences regarding one item.

The advantage of our approach is that users are not required to

redefine their preference as in Kalech et al. (2011) and are not re-

uired to hold a strict set of ordered untied preferences as in Lu

nd Boutilier (2013) . Also, in previous research the Majority strat-

gy is used for preference aggregation. To the best of our knowl-

dge, the issue of preference elicitation and returning one or more

tems under the Least Misery strategy has not yet been investi-

ated, although Least Misery is favored by users ( Masthoff, 2004 ).

urthermore, we have not encountered any study that approxi-

ates a winner or k alternative winner with a confidence level. We

resent algorithms which can use the Majority or the Least Misery

trategy in order to output one or top- k definite winner items or

pproximate winner items within some confidence level.

. Model description

We introduce a general approach for reaching a joint decision

ith minimal elicitation of voter preferences. We assume that the

sers’ preferences are unknown in advance, but can be acquired

uring the process, i.e., a user who is queried for her preference for

n item will answer the query and rate the item in question. We

lso assume that users submit their true preferences. Therefore, in

his paper we do not handle manipulations or strategic voting.

.1. Problem definition

Let us define a set of users (voters) as V = { v 1 , v 2 , . . . , v m

} and

set of candidate items as C = { c 1 , c 2 , . . . , c n } . We define a request

or specific information from the voter as a query q . A query has a

ost, e.g., the cost of communication with the voter, or the cost of

nterfering with the voter’s regular activities.

efinition 1. (Cost): Given a query q ∈ Q , the function cost : Q → returns the communication cost of the query .

L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261 249

Fig. 1. Model description.

q

r

a

e

q

c

t

v

a

e

o

o

w

s

m

D

r

a

w

t

3

o

o

i

l

i

T

t

c

c

i

q

s

c

p

t

c

b

T

t

d

k

k

c

h

b

v

e

i

S

(

t

e

i

D

a

d

o

a

i

o

Throughout this paper, we assume that the cost is equal for all

ueries. It is possible to determine the winner from partial voters’

atings ( Konczak & Lang, 20 05; Walsh, 20 07 ). We adopt an iter-

tive method ( Kalech et al., 2011 ), which proceeds in rounds. In

ach round one voter is queried for her rating of one item. Conse-

uently, we determine the next query, such that the total expected

ost is minimized.

Let O

i represent the set of voter v i ’s responses to queries. Note

hat this set does not necessarily contain all the items, since the

oter might have not been queried regarding some of the items,

nd therefore has not rated them. All the preferences of all the vot-

rs are held in O

A ; O

A = { O

1 , . . . , O

m } is a set of O

i sets. At the end

f each round, one query response is added to O

i . When the set

f voters’ preferences O

A contains enough (or all) preferences, the

inning item can be computed using some preference aggregation

trategy. Our goal is to guarantee or approximate the winner with

inimal cost.

efinition 2. (Winner with minimal cost problem): Let O

A 1 , ..., O

A s

epresent possible sets of queries and let R

A 1 , ..., R

A s represent the

ggregated cost of the queries in each set respectively. A winner

ith minimal cost is the winner determined using the set with

he lowest cost: argmi n i ( R

A i ) .

.2. The framework

We assume the existence of a voting center whose purpose is to

utput items given the termination conditions: top- k items where

ne of the items is the winner with 1 − ∝ confidence level. These

tems are presented to the group as their best options.

The complete process is illustrated in Fig. 1 and proceeds as fol-

ows: A distribution center holds a database of historical user and

tem ratings, and the users’ social networks graph if it is available.

hese ratings (with the possible addition of the graph) are used

o compute voter-item probability distributions. A voting center re-

eives the voter-item probability distributions from the distribution

enter and is responsible to execute one of the heuristics described

n Section 4 . The heuristics outputs a voter-item query, i.e., a re-

uest for one specific voter to rate one specific item. This query is

ent to the appropriate voter. Once the voter responds, the voting

enter checks whether there are enough preferences to stop the

rocess. This is done by examining the value of the threshold for

ermination. If reached, the process terminates. If not, the voting

enter resumes charge. The user’s response is sent to the distri-

ution center so that the probability distributions can be updated.

he voting center outputs a new query and the process repeats un-

il the termination condition is reached. The termination condition

epends on the preset confidence level 1 − α (0 ≤ α ≤ 1) and on

, the number of outputted items. For example, when ∝ = 0 and

= 1 , the process terminates once one definite item is found.

When queried, the voters assign ratings to the items from a dis-

rete domain of values D where d min and d max are the lowest and

ighest values, respectively. User v i ’s preferences are represented

y the rating function value : V × C → D .

We assume that there exists a prior rating distribution of the

oter-item preferences, i.e., a prior distribution of each voter’s pref-

rences for each item. The distribution can be inferred from rank-

ngs of similar users using collaborative filtering methods ( Koren &

ill, 2011 ) or using the user’s social networks graph, if such exists

Ben-Shimon et al., 2007 ). In order to decide on the next query,

he elicitor considers the rating distribution of the voter-item pref-

rences. We denote q i j

as the single preference of v i for a single

tem c j . The rating distribution is defined as follows:

efinition 3. (Rating Distribution): the voting center considers q i j

s a discrete random variable distributed according to some rating

istribution v d i j , such that v d i

j [ d g ] ≡ P r ( q i j = d g ) .

The example presented in Table 1 shows the rating distribution

f three voters for two items in the domain D = { 1 , 2 , 3 } . For ex-

mple, the probability that v 1 will assign a rating of 1 to item c 1 s 0.2.

We assume independence between the probability distributions

f each voter. While the independence assumption is naive, it can

250 L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261

Table 1

Rating distribution of the voters in the set V =

{ v 1 , v 2 , v 3 } . v 1 v 2 v 3

c 1 c 2 c 1 c 2 c 1 c 2

d 1 = 1 0 .2 0 .2 0 .4 0 .5 0 .3 0 .7

d 2 = 2 0 .2 0 .2 0 .3 0 .2 0 .3 0 .1

d 3 = 3 0 .6 0 .6 0 .3 0 .3 0 .4 0 .2

Table 2

Three users, and their preferences for 3 items.

Candidate Items

Group Members (Users) c 1 c 2 c 3

v 1 5 3 1

v 2 4 3 5

v 3 2 3 4

Majority Score 11 9 10

Least Misery Score 2 3 1

f

a

3

s

t

t

t

t

a

k

i

l

t

t

s

a

g

D

a

c

w

c

D

a

d

w

t

?

b

e

j

d

i

d

3

e

2

K

s

a

a

1 Konzak and Lang ( Konczak & Lang, 2005 ) affirm this definition of necessary and

possible winners in proposition 2 of their paper.

be used for approximating the actual probability. An attempt to ad-

dress dependency will yield probabilities that are too complex for

a system to realistically hold, since each distribution may depend

on each of the other distributions. When facing the tradeoff be-

tween the model’s accuracy and practicality, we chose to model a

practical system. However, note that the precise probability value

is not required if the queries are still sorted correctly according to

the value of the information they hold (their informativeness). In

the machine learning field, a similar naive assumption is known

to provide accurate classification, even though the independence

assumption is not always true ( Domingos & Pazzani, 1997 ). We

therefore argue that the system’s loss of accuracy, if at all exists,

is insignificant.

The initial estimated rating distribution is calculated in advance

( Koren & Sill, 2011 ). The estimated distribution is then updated ev-

ery time a new rating is added. The accuracy is expected to grow

with the number of ratings acquired.

In the next sections we present the aggregation strategies

( Section 3.3 ), the termination conditions ( Section 3.4 ) and the elic-

itors ( Section 4 ).

3.3. Aggregation strategies

The item’s score depends on the strategy used. Throughout this

paper, we denote the employed aggregation strategy str . We now

define the aggregation strategies: Majority and Least Misery. As

mentioned, in the Majority strategy the emphasis is towards the

majority of the group:

Definition 4. (Majority Strategy): given the users’ preferences,

the Majority Strategy computes the score of item c j as follows:

s ma jority ( c j ) =

∑

i ∈{ 1 , ... ,m } q i j In the Least Misery strategy, the chosen item cannot be the

least preferred by any of the users.

Definition 5. (Least Misery Strategy): given the users’ preferences,

the Least Misery Strategy computes the score of item c j as follows:

s least ( c j ) = min

i ∈{ 1 , ... ,m } q i j

Each of the two strategies has its pros and cons. The choice

of the strategy might impact the result. Consider the example in

Table 2 , showing the preferences of three users for 3 items. Ac-

cording to the Majority strategy, the winning item is item c 1 , with

a total score of 11, followed by items c 3 and c 2 . According to the

Least Misery strategy, the winning item is c , with a score of 3,

2

ollowed by items c 1 and c 3 . Therefore: st r ma jority = ( c 1 � c 3 � c 2 )

nd st r least = ( c 2 � c 1 � c 3 ) .

.4. . Termination conditions

During the preference elicitation process, the preferences are

ubmitted to the voting center and the voting center aggregates

he preferences. This process continues until a termination condi-

ion is reached. The termination condition is pre-set by the sys-

em administrator according to the group’s request. The termina-

ion condition is one of the following: a definite winning item,

n approximate winning item with some confidence level, top-

items where one of them is the winner, or approximate top- k

tems where one of the items is the winner with some confidence

evel.

Given a set of responses to queries and a termination condi-

ion, the goal is to determine whether the iterative process can be

erminated. Let O

i = { q i p , . . . , q i t } represents the set of voter v i ’s re-

ponses to queries. Note that this set does not necessarily contain

ll the items. O

A = { O

1 , . . . , O

m } is a set of O

i sets. The function

pma x A ( c j, O

A ) computes the possible maximum rating for item c j ,

iven the known preference values of the voters.

efinition 6. (Possible Maximum): given the set of responses O

A

nd an aggregation strategy str, the possible maximum score of

andidate c j , denoted pma x A ( c j , O

A ) , is computed as follows:

pma x A ( c j , O

A , str ) =

{∑

i ∈{ 1 ,...,m } pma x i ( c j , O

i ) st r =ma jorit y

mi n i ∈{ 1 ,...,m } ( pma x i ( c j , O

i ) ) str = least misery

here pma x i ( c j , O

i ) =

{d g i f q i

j = d g

d max otherwise

Similarly, the function of the possible minimum rating of item

j : pmin A ( c j , O

A ) is:

efinition 7. (Possible Minimum): given the set of responses O

A

nd an aggregation strategy str, the possible minimum score of can-

idate c j , denoted pmi n A ( c j , O

A ) is computed as follows:

pmi n A ( c j , O

A , str ) =

{∑

i ∈{ 1 ,...,m } pmi n i ( c j , O

i ) str = ma jority

mi n i ∈{ 1 ,...,m } ( pmi n i ( c j , O

i ) ) str = least misery

here pmi n i ( c j , O

i ) =

{d g i f q i

j = d g

d min otherwise

Consider the example given in Table 1 , but assume that

he rating of c 1 for v 3 is unknown: thus, q 1 1

= 5 , q 2 1

= 4 , q 3 1

= . Using Definitions 6 and 7 we can compute the possi-

le maximum and minimum under each aggregation strat-

gy. For example, the possible maximum of c 1 in the Ma-

ority strategy is pma x 1 ( c 1 , O

1 , ma jority ) = 5 + 4 + 5 , since q 3 1

= max = 5 . For the Least Misery strategy the possible max-

mum is: pma x 1 ( c 1 , O

1 , least misery ) = min ( 5 , 4 , 5 ) since q 3 1

= max = 5 . The possible minimum for the two strategies is

pmi n 1 ( c 1 , O

1 , ma jority ) = 5 + 4 + 1 and

pmi n 1 ( c 1 , O

1 , least misery ) = min ( 5 , 4 , 1 ) since q 3 1

= d min = 1 .

.4.1. Selection among top- k alternatives

One possible termination condition is to stop the preference

licitation process once at least one winner is found ( Kalech et al.,

011; Lu & Boutilier, 2013; Naamani-Dery et al., 2014 ). We follow

alech et al. (2011); Lu and Boutilier (2011) and define a neces-

ary winner set ( NW ) as a set of items whose possible minimum

ggregated rating is equal or greater than the possible maximum

ggregated rating of all the others 1 :

L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261 251

D

N

t

i

j

t

s

r

o

i

i

t

t

a

w

s

p

D

w

t

w

c

e

s

s

N

s

i

3

e

i

n

p

t

c

o

S

a

w

D

d

t

p

P

s

p

s

P

P

r

P

j

d

0

i

m

v

b

P

I

0

0

P

v

D

a

p

t

m

i

w

n

t

t

D

p

i

4

h

4

g

efinition 8. (Necessary Winners Items Set):

W = { c i | pmi n

A (c i , O

A )

≥ pma x A (c j , O

A ) ∀ c j ∈ C\ ci }

We assume that there is only one necessary item, although

he necessary winner set may theoretically contain more than one

tem. Note that for a big amount of voters, in most cases there is

ust one winner item. In the case of more than one winning item,

he first item is selected lexicographically.

In some cases, the group members can be satisfied with a

horter preference elicitation process. They may agree to trade the

esult accuracy with less elicitation cycles. In other words, instead

f terminating the preference elicitation once a necessary winner

s found, the group may agree to terminate the preference elic-

tation once a set of top- k items is presented to them. One of

hese items is the necessary winner, but without further elicita-

ion it is not possible to determine which of the items it is. To

ccurately define the top- k items, let us define first the possible

inner group. The possible winners are all the items whose pos-

ible maximum aggregated rating is greater than or equal to the

ossible minimum rating of all the other items.

efinition 9. (Possible Winners Set): P W = { c i | pma x A ( c i , O

A ) ≥pmi n A ( c j , O

A ) ∀ c j ∈ C\ ci } Note that the possible winning group subsumes the necessary

inners: NW ⊂PW . After each query, the necessary winner set and

he possible winner set need to be recalculated. To begin with,

hen none of the preferences are known, the possible winner set

ontains all items: | P W | = | C| and the necessary winner’s set is

mpty: | NW | = ∅ . The process is terminated once the size of the

et of possible winners is reduced to k . We denote the neces-

ary winners set of size k and the possible winners set of size k:

W

k and PW

k respectively. Thus, the set contains the top- k pos-

ible winners, where, by definition, these top- k are guaranteed to

nclude the necessary winners.

.4.2. Winner approximation

We examine the accuracy-elicitation tradeoff. The preference

licitation process can be reduced, but the accuracy of the output

s affected: the returned items are estimated to contain the win-

ing item at some confidence level, with an error rate α. To com-

ute a winner with some confidence level we should first define

he score space of the aggregation. The score s that the candidate

an achieve after aggregating the preferences of the voters depends

n the strategy:

=

{{ n · d min , n · d min + 1 . . . , n · d max } i f st r = ma jorit y

{ d min , d min + 1 . . . , d max } i f str = least (1)

Let us begin by examining the probability that one item has

certain score: P r( c j = s ) . The probability of any item to be the

inner is:

efinition 10. (Item Winning Probability): Under the indepen-

ence of probabilities assumption, the probability that item c j is

he winner is the aggregation of c j ’s probabilities to win over the

ossible ratings s :

r (c j ∈ NW

)=

∑

s ∈ S, ∀ i = j P r

(c j = s | v 1 , . . . , v m

)· Pr ( c i < s )

=

∑

s ∈ S∀ i = j P r

(c j = s | v 1 , . . . , v m

)·∏

∀ i = j P r( c i < s )

The probability that given m voters an item will receive the

core s P r( c j = s | v 1 , . . . , v m

) can be computed recursively. This

robability depends on the aggregation strategy. For the Majority

trategy we use:

r (c j = s | v 1 , . . . , v m

)=

d max ∑

x = d min

(P r

(c j = s − x | v 1 , ..., v m −1

)

·P r (q j m

= x ))

where P r ( c j = s | v i ) = P r( q j i = s ) (2)

For the Least Misery strategy we use:

r (c j = s | v 1 , . . . , v m

)

=

d max ∑

x =s

(P r

(c j = s | v 1 , ..., v m

)· P r

(q j m

= x ))

+

d max ∑

x =s+1

(P r

(c j = x | v 1 , ..., v m

)· P r

(q j m

= s ))

(3)

In both strategies we compute the probability that an item will

eceive a score of at most s as follows (s is defined in Eq. (1 )):

r (c j < s

)=

s −1 ∑

x = min ( S )

P r (c j = x | v 1 .. v m

)(4)

The following is a step by step running example, for the Ma-

ority strategy for d = { 1 , 2 , 3 } . The example is based on the voting

istributions (VD’s) presented in Table 1 ; note that P r( q 3 1

= 3 ) = . 4 , P r( q 3

1 = 2 ) = 0 . 3 , P r( q 3

1 = 1 ) = 0 . 3 . We start by calculat-

ng P r( c j = s ) . The calculation is done using a dynamic program-

ing algorithm where each result is calculated using the pre-

iously calculated results. For instance, using Eq. (2) , P r( c 1 = 6 )

ased on the ratings of voters v 1 , v 2 , v 3 :

P r( c 1 = 6 | v 1 .. v 3 ) = P r( c 1 = 5 | v 1 , v 2 ) × P r( q 3 j

= 1 ) +

r( c 1 = 4 | v 1 , v 2 ) × P r( q 3 j

= 2 ) + P r( c 1 = 3 | v 1 , v 2 ) × P r( q 3 j

= 3 ) .

n the same manner: P r( c 1 = 5 | v 1 , v 2 ) = 0 . 14 , P r( c 1 = 4 | v 1 , v 2 ) = . 36 , P r( c 1 = 3 | v 1 , v 2 ) = 0 . 24 so that finally P r( c 1 = 6 | v 1 .. v 3 ) = . 236 . Next, we calculate Pr ( c 1 ≤ s ) using Eq (4) : P r( c 1 < 6 ) = r( c 1 = 3 ) + P r( c 1 = 4 ) + P r( c 1 = 5 ) .

To define top- k with a confidence level we define first PV as a

ector of items, ordered according to their winning probability:

efinition 11. (Ordered Vector of winning probabilities): PV is an

rray of decreasingly ordered items according to their winning

robabilities .

The probability that the winner is within the top- k is actually

he aggregated winning probabilities of the first k items in PV . The

ore preferences elicited from the users the higher the probabil-

ty the winner is within the top- k . The confidence level is a value

hich determines an upper bound for the probability of the win-

er to be among the top- k . The preference elicitation process is

erminated once the confidence level equals 1 − α. Formally, the

ermination condition is:

efinition 12. (Termination with top- k approximate items): the

reference elicitation process terminates for a given k and α, whenk ∑

=1

P V [ i ] ≥ 1 − α where 0 ≤ α ≤ 1 .

. Elicitors

The elicitor selects the next query according to one of two

euristics, DIG or ES.

.1. Entropy based heuristic

The Dynamic Information Gain ( DIG ) Heuristic is an iterative al-

orithm ( Naamani-Dery, Kalech, Rokach, & Shapira, 2010 ). It uses

252 L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261

(

i

m

a

o

h

m

t

c

c

v

E

E

E

s

i

t

t

t

a

i

5

(

t

5

a

5

q

S

t

t

t

m

t

g

m

a greedy calculation in order to select a query out of the possi-

ble m × n queries. The chosen query is the one that maximizes

the expected information gain. The expected information gain of

a specific query is influenced by the difference between the prior

and the posterior probability of the candidates to win given the

possible responses to the query. The algorithm terminates once a

winner is within the requested top- k items with a confidence level

of 1 − ∝ . In order to select a query, the information gained from

each one of the optional queries is calculated and then the query

one that maximizes the information gain is selected. To compute

the information gain, the winning probability of each item is cal-

culated. Next, the information gain of the m × n possible queries

is computed. The expected information gain of a query is the dif-

ference between the prior expected entropy and the posterior ex-

pected entropy, given the possible responses to the query. The en-

tropy of the necessary winner within top- k ( NW

k ) is computed as

follows:

H

(N W

k )

= −n ∑

j=1

P r (c j ∈ N W

k )

· log (P r

(c j ∈ N W

k ))

Definition 13. (Information Gain): The Expected Information Gain

(IG) of a query q i j

is:

IG ( N W

k | q i j ) = H( N W

k ) −max ∑

g= min

H(N W

k | q i j = d g ) · P r( q i

j = d g ) where

H( N W

k | q i j = d g ) represents the expected entropy of NW

k given the

possible values by querying voter v i on item c j .

The query that maximizes the information gain is se-

lected: argmaxI G i, j (N W

k | q i j ) . The query selection process contin-

ues until the termination condition is reached, i.e., once a winner

within top- k items is found with 1 − ∝ confidence. Note that the

termination conditions is determined by ∝ and k . However, the ter-

mination condition does not affect the systems information gain.

4.2. Expected maximum based heuristic

The highest expected heuristic (ES) score is based on the ex-

ploration vs. exploitation tradeoff ( Naamani-Dery et al., 2010 ). As

mentioned earlier, a necessary winner is an item whose possi-

ble minimum is greater than the possible maximum of the other

items. The possible maximum of an item decreases while its pos-

sible minimum increases as more information about voter prefer-

ences is revealed. Thus, an item for which no voter has yet sub-

mitted a rating has the highest possible maximum and must be

considered as a possible winner. On the other hand, such an item

also has the lowest possible minimum and cannot yet be a winner.

Therefore, for more information, we may want to explore the vot-

ers’ preferences for the items in order to determine their potential

of being a winner within top- k . Once we have enough information

about the items’ rating, we can exploit this information to further

inquire about the items that are more likely to win.

We propose a heuristic which chooses its next query by consid-

ering the item that has the possible maximum and the voter that

is expected to maximize the rating of that item. The expected rat-

ing of q i j

based on the rating distribution v d j i

is:

ES (v d i j

)=

max ∑

g= min

Pr (q i j = d g

)· d g (5)

For item c j , we choose the voter that maximizes the expected

rating: argma x i ES( v d i j ) . Using this approach, we encourage a broad

exploration of the items since the less information we have about

an item’s rating, the higher possible maximum it has. In addition,

we exploit the preferences revealed in order to: (1) refrain from

querying about items that have been proven as impossible winners

since their possible maximum is less than a minimum of another

tem); (2) further examine an item that has the highest possible

aximum and might be a necessary winner.

The following is an illustration of the algorithm using the ex-

mple employed in the previous section. To begin with, we have

nly probabilistic knowledge of voter preferences. Since no voter

as submitted any preference yet, in the first round the possible

aximum of each item is nine (since there are three voters and

he maximum rating that can be assigned is three). The first item

1 is selected for a query according to the tie breaking policy. Ac-

ording to the distribution in Table 1 , the expected ratings of the

oters over c 1 are:

S (v d 1 1

)= 0 . 2 · 1 + 0 . 2 · 2 + 0 . 6 · 3 = 2 . 4

S ( v d ) = 0 . 4 · 1 + 0 . 3 · 2 + 0 . 3 · 3 = 1 . 9

S (v d 1 3

)= 0 . 3 · 1 + 0 . 3 · 2 + 0 . 4 · 3 = 2 . 3

Thus, the voter-item query pair is q 1 1 . Assuming the voter’s re-

ponse is q 1 1 = 2 , in the next iteration the possible maximum of c 1 s 8 and of c 2 is 9. Therefore, in the next round, c 2 is selected as

he item in the voter-item query pair. The algorithm iterates until

he termination condition is reached. For instance, if the termina-

ion condition is an approximate item with α = 0 . 05 and k = 3 , the

lgorithm will repeat until a necessary winner is one of the top-3

tems, with a probability of 95%.

. Evaluation

We first present the research questions and research procedure

Section 5.1 ), the datasets evaluated ( Section 5.2 ) and then present

he evaluation on: different top- k termination conditions ( Section

.3 ), different confidence levels for approximation ( Section 5.4 ),

nd a comparison of the two aggregation strategies ( Section 5.5 ).

.1. Research questions and research procedure

We present an empirical evaluation of the following research

uestions:

(a) Selection –To what extent does outputting top- k items re-

duce the required number of queries ( Section 5.3 )?

(b) Approximation – there is a tradeoff between outputting an

approximate winner, or approximate top- k items and out-

putting a definite winner or definite top- k items. To what

extent does the approximation accuracy improve as more

data is collected ( Section 5.4 )?

(c) Aggregation – How does the aggregation strategy affects the

preference elicitation process? We examine two aggregation

strategies: with emphasis towards the group and with em-

phasis towards the user; i.e., the Majority and Least Misery

strategies ( Section 5.5 ).

We examine the performance of the algorithms presented in

ection 4 : DIG and ES. As mentioned in the related works sec-

ion, to the best of our knowledge, there are no other algorithms

hat operate (or can be expanded to operate) under the same set-

ings. Therefore, the baseline for measuring the effectiveness of our

ethod is a random procedure (RANDOM), which randomly selects

he next query. To account for the randomness of the RANDOM al-

orithm each experiment is repeated 20 times. We evaluate the

ethods in terms of:

(1) Number of queries - we measure the reduction in commu-

nication cost by measuring the number of queries required

for finding a necessary or approximate winner.

(2) Approximation accuracy – the approximation accuracy is

measured in two ways:

(a) Probability that the winner is within the top- k

L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261 253

t

p

u

h

l

h

5

b

1

g

i

i

h

m

t

w

e

W

o

n

t

f

g

h

5

a

S

s

d

n

t

i

d

s

u

r

t

(

a

A

s

Table 3

Skewness levels.

Skewness level d 1 = 1 d 2 = 2 d 3 = 3 d 4 = 4

−6 0 .011 0 .011 0 .147 0 .832

0 0 .25 0 .25 0 .25 0 .25

6 0 .832 0 .147 0 .011 0 .011

s

i

r

f

r

5

w

d

s

t

a

u

d

w

d

d

s

r

t

T

n

s

f

5

i

a

s

B

(

o

a

i

p

s

i

c

t

(

5

1

w

j

D

m

(b) Confidence level accuracy - a confidence level refers to

the percentage of all possible samples that can be ex-

pected to include the true population parameter. The

confidence level ( 1 − ∝ ) is accurate, if the winner is in-

deed within the top- k items in ( 1 − ∝ )% of the experi-

ments.

In order to conclude which algorithm performs best over mul-

iple datasets, we follow a robust non-parametric procedure pro-

osed by ( García, Fernández, Luengo, & Herrera, 2010 ). We first

sed the Friedman Aligned Ranks test in order to reject the null

ypothesis that all heuristics perform the same. This test was fol-

owed by the Bonferroni-Dunn test to find whether one of the

euristics performs significantly better than other heuristics.

.2. Datasets

We evaluated our methods on different domains:

• Simulated datasets : allowed us to manipulate the data and

thus further study the different parameters. • Real-world datasets : the Netflix data ( http://www.netflixprize.

com ), Sushi dataset ( Kamishima et al., 2005 ). • User Study datasets : Pubs dataset and Restaurants dataset

datasets from the “Lets do it” 2 recommender system, a user

study performed in Ben Gurion University during Spring 2014.

In each domain we considered a setting of a group of 10 mem-

ers and 10 items. We chose to restrict the users-items matrix to

0 ×10 since we tried to model a common scenario of a small

roup of people that wish to reach a joint decision. It would be

mpractical to suggest an elicitation process on large numbers of

tems (e.g., 16, 0 0 0 movies). Even for less extreme cases, studies

ave shown that too much choice can be demotivating. Users are

ore satisfied when presented with a single-digit number of op-

ions to choose from Iyengar and Lepper (20 0 0) . Therefore in cases

here a large number of items exist, we suggest to apply a prefer-

nce elicitation voting procedure on the top N ranked items only.

e assume that when more than N items are available, some rec-

mmender system can be used to provide a ranked list of all items.

Our focus is the analysis of the contribution of returning a win-

er within top- k items, thus narrowing down the top- N sugges-

ions received by a recommender system ( k ≤ N ). An additional

ocus is on approximating a winner and on the aggregation strate-

ies. The analysis of the scaling of the matrix sizes and the runtime

as been evaluated in Naamani-Dery et al. (2014) .

.2.1. Simulated datasets

The first domain is a simulated meeting scenario where voters

re required to vote for their preferred time slot for a meeting.

imulating the data allows us to investigate different distribution

ettings and a wider variety of scenarios than those given in one

ataset. The users rate their preferences on a scale of 1–4. We ma-

ipulated the user-item distribution skewness, i.e, the measure of

he asymmetry of a distribution. A higher absolute skewness level

ndicates a higher asymmetry. A negative skew indicates that the

istribution is concentrated on high vote values while a positive

kew indicates the distribution is concentrated on low vote val-

es. Similarly to Naamani-Dery et al. (2014) , we created user-item

ating distributions with different skewness levels. We chose 3 ex-

reme cases, as presented in Table 3: a user is in favor of the item

skewness level ( −6)), a user dislikes the item (skewness level 6),

user is indifferent (skewness level 0). In the experiments, the

2 The “Lets Do It” recommender systems was developed by Eli Ponyatovski and

viad Carmeli, 4th year students in the Information Systems Department, under the

upervision of: Lihi Dery, Ofrit Lesser and Meir Kalech, Ben Gurion University 2014.

p

l

d

F

kewness level of one of the items is set in advance and all other

tems receive a uniform skew (skew “0” in Table 3 ). Having set a

ating distribution for every user-item pair, we randomly sample

rom the distribution to set the voter-item rating. To account for

andomness, each experiment was repeated 10 times.

.2.2. Real-world datasets

The Netflix prize dataset ( http://www.netflixprize.com ) is a real

orld dataset containing the ratings voters assigned to movies. The

ataset consists of ∼10 0,0 0 0 users and ∼16,0 0 0 items. We con-

ider a setting of a group of 10 users and 10 items. The users in

he group were drawn randomly from a subset of Netflix where

ll of the users rated 100 items and there were no missing val-

es. 90 items were used to create the initial rating distribution as

escribed in Naamani-Dery et al. (2014) . The 10 remaining items

ere set as the items in question. To account for Randomness, 10

ifferent groups were extracted in this manner.

The Sushi dataset ( Kamishima et al., 2005 ) is a real world

ataset that contains 50 0 0 preference rankings over 10 kinds of

ushi. 10 different groups of 10 different users each were drawn at

andom from the dataset. Since only 10 items exist in the dataset,

he initial user-item probability distribution was set to uniform.

he distribution was updated after each query. We examined a sce-

ario of 10 users who have to decide between 10 sushi types, using

ubsets of the dataset. We derived 10 different random matrices

rom each scenario size.

.2.3. User study

We created our own set of real data and examined two scenar-

os of a group that wishes to: (a) select a restaurant or (b) select

pub or club. The data was collected using a group recommender

ystem, named “Lets Do It”.

The system obtained a full set of ratings from 90 students in

en Gurion University, for two different domains: (a) restaurants

16 items) and (b) pubs and clubs (23 items). Fig. 2 presents the

pening screen. The students were instructed to rate each item on

1 to 5 scale, according to their satisfaction from past visits, or

n case they were unfamiliar with a place, according to how ap-

ealing it was for them to visit it. Each item had a picture and a

hort description, as shown in Fig. 3 . The students could view the

tems they rated, the items left for them to rate. They could also

hange the ratings. This is demonstrated in Fig. 4 . Rating distribu-

ions were derived in the same manner as for the Netflix dataset

Section 5.2.1 ).

.3. Selection of top- k items

We examined different top- k termination conditions, from k = (i.e., requiring one definite winner), to k = 9 (i.e., requiring the

inner to be one of the top-9 items). The results are for the Ma-

ority aggregation strategy with a 100% confidence level ( ∝ = 0) .

ifferent confidence levels and a comparison between the perfor-

ance of the Majority strategy and the Least Misery strategy are

resented in the next sections. We first report the results of three

evels of skewness of simulated data, followed by the real world

atasets: Netflix, Sushi, Pubs, and Restaurants.

We examine three different skewness levels of simulated data.

igs. 5 –7 present results for a skewness level of (6), (0) and ( −6)

254 L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261

Fig. 2. The student rate pubs&clubs and restaurants.

Fig. 3. Rating for two clubs.

Fig. 4. The student can see what places need to be rate.

L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261 255

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

Num

ber o

f qu

erie

s

top-k

Fig. 5. Heuristics comparison for top- k with skewness level ( −6).

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

DIG

ES

RANDOM

Fig. 6. Heuristics comparison for top- k with skewness level (0).

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

Fig. 7. Heuristics comparison for top- k with skewness level (6).

r

A

n

t

d

T

n

E

s

f

v

fl

s

a

w

t

c

(

a

D

9

t

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

Fig. 8. Heuristics comparison for top- k on the Netflix dataset.

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

Fig. 9. Heuristics comparison for top- k on the Sushi dataset.

0

50

100

150

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

Fig. 10. Heuristics comparison for top- k on the Pubs dataset.

0

50

100

150

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

Fig. 11. Heuristics comparison for top- k on the Restaurants dataset.

n

m

n

w

f

r

f

i

t

c

espectively. Axis x presents the termination conditions k = 1 , .., 10 .

xis y presents the amount of queries required in order to termi-

ate and find a winner within the top- k . A larger k means that the

ermination condition is relaxed and less queries are needed. In-

eed, in all cases, as k increases, the amount of queries decreases.

he performance of RANDOM is not significantly affected by skew-

ess levels. For a skewness level of −6 ( Fig. 5 ), DIG outperforms

S and RANDOM and requires the least amount of queries. For a

kewness level of (0) and of (6), ES outperforms DIG and RANDOM

or the top-1 to top-3 items. Then, DIG resumes charge and pro-

ides better results ( Figs. 6 and 7 ).

We now turn to examine the real world datasets. On the Net-

ix dataset ( Fig. 8 ), the trend is similar to that obtained on the

kewness level of 0 and 6. That is, for top-1 to top-3 ES is superior,

nd then DIG maintains the lead. Again, DIG displays a sharp curve

hile ES requires almost the same number of queries regardless of

he termination point (the top- k ). The same phenomenon is radi-

alized on the Pubs dataset ( Fig. 10 ) and on the Restaurants dataset

Fig. 11 ); not only does DIG take the lead after the top-3 items, but

lso, Random exhibits better performance than ES (but not than

IG) when for more than top-6 items. On the Sushi dataset ( Fig.

) DIG outperforms ES and RANDOM for all k .

The results can be explained by considering the properties of

he heuristics and of the datasets. In a setting of a simulated skew-

ess of ( −6) the votes are skewed towards the winner and it is

ore obvious who the winner is. It is less obvious who the win-

er is when the skewness level is 0 or 6 in simulated data. Also,

hen k is smaller, ES performs better, since ES is designed to seek

or potential winning items. Therefore, the amount of queries ES

equires is more or less constant regardless of the k items required

or output. DIG is designed to focus on reducing entropy. When k

s larger the entropy reduces faster. In the Sushi dataset the ini-

ial user-item distribution is uniform so all items have the same

hance to be the winning item. Thus, the initial state in the Sushi

256 L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261

0

20

40

60

80

100

0.5 0.6 0.7 0.8 0.9 1

Num

ber o

f que

ries

Confidence level

Fig. 12. Approximations with simulated data with skewness (0).

0

20

40

60

80

100

0.50 0.60 0.70 0.80 0.90 1.00

Num

ber o

f que

ries

Confidence level

DIGESRANDOM

Fig. 13. Approximations on the Netflix dataset.

0

20

40

60

80

100

0.50 0.60 0.70 0.80 0.90 1.00

Num

ber o

f que

ries

Confidence level

Fig. 14. Approximations on the Sushi dataset.

0

50

100

150

0.50 0.60 0.70 0.80 0.90 1.00

Num

ber o

f que

ries

Confidence level

Fig. 15. Approximations on the Pubs dataset.

0

50

100

150

0.50 0.60 0.70 0.80 0.90 1.00

Num

ber o

f que

ries

Confidence level

Fig. 16. Approximations on the Restaurants dataset.

0

0.2

0.4

0.6

0.8

1

1 7 13192531374349556167Prob

abili

ty a

win

ner

is

wit

hin

top-k

Itera�on number

Fig. 17. Simulated data: the probability the winner is within top- k .

q

N

h

t

(

t

n

s

a

a

dataset is similar to a simulated skewness data with (0). However,

in the Netflix, Pubs, and Restaurants datasets the distributions are

estimated and there is a skewness pattern which enables DIG to

outperform. Furthermore, when it is less obvious who the winner

is (as in Netflix), the differences in the heuristics performance are

smaller.

For all datasets, the Friedman Aligned Ranks test with a con-

fidence level of 95% rejects the null-hypothesis that all heuris-

tics perform the same. The Bonferroni-Dunn test concluded that

DIG and ES significantly outperform RANDOM at a 95% confidence

level.

5.4. Approximation

We examined the amount of queries required under different

confidence levels ( Figs. 12 and 13 ), when a definite winner ( k = 1)

is required. For the simulated data, we set the skewness level to

neutral (0). The results presented here are for the Majority strat-

egy, while a comparison between the two aggregation strategies is

presented in the next section. We also examine the accuracy of the

approximations.

Axis x presents the required confidence level; from 50% to 100%

(100% is a definite winner). Axis y presents the amount of queries

required in order to terminate and find the top- k items. For the

simulated data, there is a steady increase in the required amount

of queries ( Fig. 12 ) for all heuristics. DIG outperforms ES and RAN-

DOM, while RANDOM is the least performer ( Fig. 14 ). The steady

increase in the amount of queries for the simulated dataset and

for the Sushi dataset ( Kamishima et al., 2005 ), Pubs dataset ( Fig.

15 ) and Restaurants dataset ( Fig. 16 ) can be easily explained since

more queries are needed in order to gain more information for a

higher accuracy level. However, the results for the Netflix dataset

behave differently and require a deeper explanation.

For the Netflix data ( Fig. 13 ), the increase in the required

amount of queries is small for confidence levels 50%–95%. How-

ever, there is a big jump in the required number of queries when

the desired confidence is 100% (a definite winner is required): from

∼10 required queries to achieve a confidence level of 95%, to ∼90

ueries for a 100% confidence. The probability distributions for the

etflix dataset are estimated, whereas for the simulated data we

ave accurate (simulated) distributions. We show the probabili-

ies accuracy for the datasets: simulated data with skewness level

0), Netflix and Sushi in Figs. 17 , 18 and 19 respectively. Axis x is

he iteration number and axis y is the probability that the win-

er is indeed within the top- k items. In this case, k = 1 . For the

imulated data ( Fig. 17 ) the probability accuracy increases steadily

s more information, acquired in the iterations, becomes avail-

ble. On the other hand, since the Netflix, Pubs and Restaurants

L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261 257

Table 4

Accuracy of DIG for different confidence levels.

Confidence level Simulated data Netflix data Sushi data Pubs data Restaurants data

0 .5 60% 50% 80% 30% 70%

0 .55 60% 50% 80% 30% 90%

0 .6 60% 50% 70% 30% 80%

0 .65 60% 50% 80% 20% 80%

0 .7 80% 50% 70% 40% 80%

0 .75 90% 50% 90% 30% 80%

0 .8 100% 60% 90% 40% 80%

0 .85 100% 60% 90% 40% 80%

0 .9 100% 60% 90% 40% 80%

0 .95 100% 70% 90% 60% 80%

0

0.2

0.4

0.6

0.8

1

1 3 5 7 9 11 13 15 17 19 21Prob

abili

ty w

inne

r is

w

ithi

n to

p-k

Itera�on number

Fig. 18. Netflix data: the probability the winner is within top- k .

0

0.2

0.4

0.6

0.8

1

1 7 13192531374349556167Prob

abili

ty w

inne

r is

w

ithi

n to

p-k

Itera�on number

Fig. 19. Sushi data - probability winner is within top- k .

p

a

b

c

e

a

t

fi

t

D

l

s

p

l

w

m

p

d

T

b

a

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top k

Fig. 20. DIG with Majority (MAJ) strategy different skewness levels.

0

20

40

60

80

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

DIG_LM -6

DIG_LM 0

DIG_LM 6

Fig. 21. DIG with Least Misery (LM) strategy different skewness levels.

0

20

40

60

80

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top k

Fig. 22. ES with Majority (MAJ) strategy different skewness levels.

e

t

i

5

e

a

6

robabilities are estimations, there is more noise until a 95% prob-

bility is reached ( Fig. 18 ). The Sushi dataset also contains proba-

ility estimations, but the estimation is more accurate ( Fig. 19 ). To

onclude, when the probability estimation is accurate, there is lin-

ar relationship between the number of required queries and the

pproximation level. However, an inaccurate probability distribu-

ion results in a “jump” when the required confidence is a 100%.

For all datasets, the Friedman Aligned Ranks test with a con-

dence level of 95% rejected the null-hypothesis that all heuris-

ics perform the same. The Bonferroni–Dunn test concluded that

IG and ES significantly outperform RANDOM at a 95% confidence

evel.

Another interesting question is whether the confidence level re-

ults are accurate. A confidence level refers to the percentage of all

ossible samples that can be expected to include the true popu-

ation parameter. The confidence level ( 1 − ∝ ) is accurate, if the

inner is indeed within the top- k items in ( 1 − ∝ )% of the experi-

ents. We analyzed the accuracy for the DIG algorithm (since it

roved to be the best algorithm for approximation settings) for

ifferent confidence levels for k = 1 . The results are presented in

able 4 . As previously shown, since the estimation of the proba-

ility distribution of Netflix, Pubs and Restaurants datasets is less

ccurate, the results for Netflix are less accurate. The accuracy is

ffected by the bias in the user rating and is beyond the scope of

his research. See Koren and Sill (2011) for further details on treat-

ng bias.

.5. Aggregation

We compared the two strategies: Majority (MAJ) and Least Mis-

ry (LM) on the DIG ( Figs. 20 and 21 ) and ES heuristics ( Figs. 22

nd 23 ) for simulated data with different skewness levels: −6, 0,

. Axis x presents the required top- k items and axis y presents the

258 L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261

Fig. 23. ES with Least Misery (LM) strategy different skewness levels.

0

20

40

60

80

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

DIG w MAJES w MAJDIG w LMES w LM

Fig. 24. DIG and ES with MAJ and LM simulated data on skewness level −6.

0

20

40

60

80

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

Fig. 25. Skewness 0.

0

50

100

150

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top k

Fig. 26. Netflix dataset: strategies comparison, top- k .

-30

20

70

120

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

Fig. 27. Sushi dataset: strategies comparison, top- k .

0

50

100

150

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

Fig. 28. Pubs dataset: strategies comparison, top- k .

0

50

100

150

1 2 3 4 5 6 7 8 9 10

Num

ber o

f que

ries

top-k

Fig. 29. Restaurants dataset: strategies comparison, top- k .

d

x

n

a

F

h

e

n

f

d

number of queries. DIG and ES with MAJ perform the same for

skewness levels 0 and 6, but it is better when the skewness is −6.

However, for the DIG and ES with LM, skewness levels have no

significant effect on the performance since skewness does not in-

dicate the quantity of low scores in the dataset, and the low scores

are exactly the issue that needs to be considered in LM.

A comparison between DIG with MAJ and DIG with LM on sim-

ulated data on skewness level 6 ( Fig. 24 ) and on skewness level

0 ( Fig. 25 ) reveals that the LM strategy outperforms MAJ in situa-

tions such as these: in a uniform skewness (skewness level 0) and

in k > 4 in skewness level -6. This can be explained by the fact

that in a setting that is not skewed towards a certain candidate

(i.e., any setting apart from −6), there might be more users that

voted “1” therefore, LM uses a tie-break to terminate. Thus, LM re-

quires fewer queries in this situation. In the Netflix dataset ( Fig.

26 ) MAJ outperforms LM, further indicating the fact that LM has

no additional value when there is no skewness towards a certain

winner. Similarly, on the Sushi dataset ( Fig. 27 ), MAJ outperforms

LM when k < 5 and then the trend changes and LM outperforms

MAJ. On the pubs and restaurant datasets ( Figs. 28 and 29 ) LM out-

performs MAJ for both heuristics. These results might be explained

by the data skewness.

We compared MAJ and LM with respect to the approxima-

tion termination condition, with a constant value of k = 1 on the

atasets: Netflix, Sushi, Pubs, and Restaurants ( Figs. 30–33 ). Axis

presents the required confidence level and axis y presents the

umber of queries. There is no significant difference between MAJ

nd LM for DIG on the Netflix, Pubs, and Restaurants dataset.

or ES, on the other hand, MAJ outperforms LM. This is since ES

euristic does not accommodate any consideration of Least Mis-

ry, as it always seeks for the item expected to win, and does

ot consider the least preferred items. The same results for ES are

ound on the Sushi dataset ( Fig. 31 ). However, for DIG on the Sushi

ataset, LM outperforms MAJ for confidence levels 50%-95%. For

L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261 259

0

50

100

150

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Num

ber o

f que

ries

Confidence Level

DIG w MAJES w MAJDIG w LMES w LM

Fig. 30. Netflix dataset: strategies comparison, approximation.

020406080

100

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Num

ber o

f que

ries

Confidence Level

Fig. 31. Sushi dataset: strategies comparison, approximation.

0

50

100

150

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Num

ber o

f que

ries

Confidence Level

Fig. 32. Pubs dataset: strategies comparison, approximation.

0

50

100

150

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Num

ber o

f que

ries

Confidence Level

Fig. 33. Restaurants dataset: strategies comparison, approximation.

c

n

j

a

t

d

p

e

t

6

n

c

l

k

o

t

α

d

e

t

n

p

v

s

m

2

a

t

e

d

fl

p

s

s

e

s

m

h

w

t

c

u

o

t

b

t

k

s

t

w

a

v

h

p

t

n

m

t

t

a

r

t

p

o

c

r

a

onfidence level 100%, MAJ outperforms LM. Namely, for one defi-

ite winner the system’s entropy can be reduced faster for the Ma-

ority aggregation strategy than for the Least Misery strategy prob-

bly since Least Misery requires more queries in order to validate

hat none of the users are miserable.

For all datasets, the Friedman Aligned Ranks test with a confi-

ence level of 90% rejected the null-hypothesis that all heuristics

erform the same for different approximation levels. We did not

xecute the Bonferroni-Dunn test since there is not one algorithm

hat is preferred over the others.

. Conclusions

We suggest considering the aggregation strategy and the termi-

ation conditions when attempting to reduce preference elicitation

ommunication cost. We examined two termination conditions: se-

ection and approximation . The first condition, selection , returns top-

items where one of them is the winning item rather than just

ne ( k = 1 ) definite winning item. The second termination condi-

ion, approximation , returns top- k items with some confidence level

(0 ≤ α ≤ 1), rather than top- k items where one of them is the

efinite winner ( α = 1) . Furthermore, we examined the Least Mis-

ry aggregation strategy and the Majority aggregation strategy.

The final goal of this paper is to employ selection, approxima-

ion and aggregation in order to reduce the amount of queries

eeded during a preference elicitation process for a group of peo-

le that want to reach a joint decision. We focused on the Range

oting protocol as it is very commonly applied for recommender

ystems. We implemented two heuristics whose primary aim is to

inimize preference elicitation: DIG and ES ( Naamani-Dery et al.,

014 ). These are the only two publicly available heuristics that aim

t reducing preference elicitation for the Range voting protocol. To

he best of our knowledge, there are no other algorithms that op-

rate (or can be expanded to operate) under the same settings.

We performed an experimental analysis on two real-world

atasets: the Sushi dataset ( Kamishima et al., 2005 ) and the Net-

ix prize dataset ( http://www.netflixprize.com ). In order to analyze

ossible skewness levels in data, we simulated data with different

kewness levels. Lastly, we examined real data collected in a user

tudy on a recommender system in Ben Gurion University. We also

stimated user-item probability distribution for all datasets.

In general, we show that selecting the suitable aggregation

trategy and relaxing the termination condition can reduce com-

unication cost up to 90%. We also show the benefits of the DIG

euristic for reducing the communication cost. In our previous

ork ( Naamani-Dery et al., 2014 ) we conclude that in most cases

he ES heuristic outperforms the DIG heuristic. The ES heuristic fo-

uses on identifying the current local maximum and queries the

ser that maximizes this item the most. The DIG heuristic focuses

n reducing the system entropy. In this paper we reveal that when

he termination conditions are relaxed, DIG takes the lead.

We examined how the number of required queries is affected

y the request to (1) return one definite winner, and (2) return

op- k items. In the latter case, the group members are left with

items to select from (selection termination condition). With re-

pect to the selection condition, there is an inverse linear connec-

ion: as k is larger the amount of required queries is reduced. Only

hen the dataset is skewed towards a certain winner item, and

lso k is set to 0 ≤ k ≤ 3, does ES outperform DIG. This obser-

ation assists to determine the conditions in which each of these

euristics should be employed. Also, we can now state that, as ex-

ected intuitively, in cases where the group members are willing

o accept a set of items rather than one winning item, the commu-

ication cost is reduced. For example, if a group’s wish to select a

ovie can be satisfied with the system offering them a choice of

op-3 movies rather than the system determining one movie for

hem, less queries to group members will be executed.

We studied (1) the tradeoff between finding the optimal winner

nd thus having an accurate result, and (2) the number of queries

equired for the process. For the approximation termination condi-

ion, we show that the amount of required queries increases pro-

ortionally to the confidence level. We show that DIG and ES can

utput accurate approximate recommendations. However, the ac-

uracy is derived from the dataset’s probability distribution accu-

acy. When the probability distribution is known or is estimated

ccurately, the recommendations are more accurate.

260 L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261

B

B

B

C

D

D

F

G

I

J

J

K

K

K

K

L

L

L

M

M

With respect to the aggregation strategy, we show that the Ma-

jority strategy does not always outperform the Least Misery strat-

egy. It is reasonable to assume that the strategy will be set ac-

cording to the users’ preferences and not according to the data.

We demonstrate the feasibility of choosing either strategy on the

datasets.

6.1. Discussion

Our findings append to a growing body of literature on pref-

erence elicitation using voting rules ( Kalech et al. 2011; Lu &

Boutilier 2011 ). Our research adds a unique contribution to pref-

erence elicitation in social choice in a number of perspectives that

have previously been overlooked. First, we have studied preference

elicitation using a non-ranking protocol (represented by the Range

protocol). Previous research has focused only on the ranking Borda

protocol. Non-ranking is worth considering since it is abundant

and often used by different applications such as netflix.com and

booking.com. Secondly, we have suggested methods for reducing

the amount of queries: (a) to return a list of top- k items where one

of them is the necessary winner; and (b) to approximate the nec-

essary winners or top- k items. These methods offer a decrease in

the required amount of queries and have not been previously sug-

gested. Finally, we examined the effect of aggregating the prefer-

ences in other strategies but the Majority based strategy. The Least

Misery strategy is often needed in real-life scenarios yet has pre-

viously been overlooked (e.g., a group looking for a dining location

may wish to avoid a fish restaurant if one of the group members

dislikes fish).

From the recommender systems domain perspective, this study

suggests a framework for preference elicitation that can be used as

a second step procedure in group recommenders: to narrow down

the predicted items list and present the group of users with def-

inite or approximate necessary winners. Group recommender sys-

tems often focus on improving the systems accuracy and usually

return a prediction to the group and not definite winning items.

A group recommender system can process thousands of candidate

items and return a list of top- N items predicted as the most suit-

able to the group. We can enhance this by eliciting user prefer-

ences on these N items and return a definite winner or top- k items

( k ≤ N ) where one of the items is the winner or an approximate

winner with some confidence level. This contribution may add to

the usability of a group recommender system offering a platform

that enables reaching a joint decision with minimal effort.

As a direct consequence of this study, we encountered a num-

ber of limitations, which need to be considered. We assumed that:

the user always provides an answer to the query, independence

of rating and equal communication cost. This can be overcome by

tweaking the model. For example, it is possible to model the prob-

ability that the user will answer the query. For a small number

of voters and items it is possible to consider dependent probabili-

ties. The communication cost be modeled as a weighted vector and

added to the model.

We examined the two aggregation strategies most common in

the literature. Extension to other available aggregation strategies

does not require a fundamental change since the heuristics and

the model do not change. We leave this for future work. Analyzing

other termination conditions is yet another promising direction to

pursue.

References

Arrow, K. J. (1951). Social choice and individual values (2nd ed. 1963). New Haven:Cowles Foundation .

Bachrach, Y. , Betzler, N. , & Faliszewski, P. (2010). Probabilistic possible winner de-termination. In Paper presented at the Proceedings of the twenty-fourth AAAI

conference on artificial intelligence (AAAI), Atlanta, GA, USA .

aldiga, K. A. , & Green, J. R. (2013). Assent-maximizing social choice. Social Choiceand Welfare, 40 (2), 439–460 .

altrunas, L. , Makcinskas, T. , & Ricci, F. (2010). Group recommendations with rankaggregation and collaborative filtering. In Paper presented at the Proceedings of

the fourth ACM conference on recommender systems (pp. 119–126) . en-Shimon, D. , Tsikinovsky, A. , Rokach, L. , Meisles, A. , Shani, G. , & Naa-

mani, L. (2007). Recommender system from personal social networks. In Ad-vances in intelligent web mastering (pp. 47–55). Springer .

Betzler, N. , Hemmann, S. , & Niedermeier, R. (2009). A multivariate complexity anal-

ysis of determining possible winners given incomplete votes. Proceedings of 21stIJCAI, 2 (3), 7 .

Brams, S. J. , & Fishburn, P. C. (2002). Voting procedures. In Handbook of social choiceand welfare: Vol. 1 (pp. 173–236) .

Chen, Y. , & Cheng, L. (2010). An approach to group ranking decisions in a dynamicenvironment. Decision Support Systems, 48 (4), 622–634 .

onitzer, V. , & Sandholm, T. (2005). Communication complexity of common voting

rules. In Paper presented at the Proceedings of the 6th ACM conference on elec-tronic commerce (pp. 78–87) .

ing, N. , & Lin, F. (2013). Voting with partial information: what questions to ask?In Paper presented at the Proceedings of the 2013 international conference on

autonomous agents and multi-agent systems (pp. 1237–1238) . omingos, P. , & Pazzani, M. (1997). On the optimality of the simple bayesian classi-

fier under zero-one loss. Machine Learning, 29 (2), 103–130 .

ilmus, Y. , & Oren, J. (2014). Efficient voting via the top- k elicitation scheme: a prob-abilistic approach. In Paper presented at the Proceedings of the fifteenth ACM

conference on economics and computation (pp. 295–312) . García, S. , Fernández, A. , Luengo, J. , & Herrera, F. (2010). Advanced nonparametric

tests for multiple comparisons in the design of experiments in computationalintelligence and data mining: experimental analysis of power. Information Sci-

ences, 180 (10), 2044–2064 .

elain, M. , Pini, M. S. , Rossi, F. , & Venable, K. B. (2007). Dealing with incompletepreferences in soft constraint problems. In Paper presented at the Proceedings of

the 13th international conference on principles and practice of constraint pro-gramming (pp. 286–300) .

Hazon, N. , Aumann, Y. , Kraus, S. , & Wooldridge, M. (2008). Evaluation of electionoutcomes under uncertainty. In Paper presented at the Proceedings of the 7th

international joint conference on autonomous agents and multiagent system-

s-Volume 2 (pp. 959–966) . yengar, S. S. , & Lepper, M. R. (20 0 0). When choice is demotivating: can one desire

too much of a good thing? Journal of Personality and Social Psychology, 79 (6),995 .

ameson, A. (2004). More than the sum of its members: challenges for group rec-ommender systems. In Paper presented at the Proceedings of the working con-

ference on advanced visual interfaces (pp. 48–54) .

ameson, A. , & Smyth, B. (2007). Recommendation to groups. In The adaptive web(pp. 596–627). Springer .

alech, M. , Kraus, S. , Kaminka, G. A. , & Goldman, C. V. (2011). Practical voting ruleswith partial information. Autonomous Agents and Multi-Agent Systems, 22 (1),

151–182 . amishima, T. , Kazawa, H. , & Akaho, S. (2005). Supervised ordering-an empirical

survey. In Paper presented at the data mining, fifth IEEE International conference(p. 4) .

onczak, K. , & Lang, J. (2005). Voting procedures with incomplete preferences. In

Paper presented at the nineteenth international joint conference on artificial in-telligence , Edinburgh, Scotland (p. 20) .

oren, Y. , & Sill, J. (2011). OrdRec: an ordinal model for predicting personalized itemrating distributions. In Paper presented at the Proceedings of the fifth ACM con-

ference on recommender systems (pp. 117–124) . ang, J. , Pini, M. S. , Rossi, F. , Venable, K. B. , & Walsh, T. (2007). Winner de-

termination in sequential majority voting. In Paper presented at the Proceed-

ings of the 20th international joint conference on artificial intelligence (IJCAI)(pp. 1372–1377) .

Lu, T. , & Boutilier, C. (2011). Robust approximation and incremental elicitation invoting protocols. In Paper presented at the IJCAI Proceedings-international joint

conference on artificial intelligence : 22 (p. 287) . u, T. , & Boutilier, C. (2013). Multi-winner social choice with incomplete prefer-

ences. In Paper presented at the Proceedings of the twenty-third international

joint conference on artificial intelligence (IJCAI-13) , Beijing (pp. 263–270) . u, T. , & Boutilier, C. E. (2010). The unavailable candidate model: a decision-theoretic

view of social choice. In Paper presented at the Proceedings of the 11th ACMconference on electronic commerce (pp. 263–274) .

Masthoff, J. (2011). Group recommender systems: combining individual models. InRecommender Systems Handbook (pp. 677–702) .

Masthoff, J. (2004). Group modeling: selecting a sequence of television items to

suit a group of viewers. User Modeling and User-Adapted Interaction, 14 (1), 37–85 .

cCarthy, J. F. , & Anagnost, T. D. (1998). MusicFX: an arbiter of group preferencesfor computer supported collaborative workouts. In Paper presented at the Pro-

ceedings of the 1998 ACM conference on computer supported cooperative work(pp. 363–372) .

yerson, R. B. (1981). Utilitarianism, egalitarianism, and the timing effect in so-

cial choice problems. Econometrica: Journal of the Econometric Society, 49 , 883–897 .

Naamani-Dery, L., Kalceh, M., Rokach, L., & Shapira, B. (2014). Reaching a joint de-cision with minimal elicitation of voter preferences. Information Sciences, 278 ,

466–487. doi: 10.1016/j.ins.2014.03.065 .

L. Naamani-Dery et al. / Expert Systems With Applications 61 (2016) 246–261 261

N

N

N

O

O

P

P

R

S

S

W

X

Y

aamani-Dery, L. , Kalech, M. , Rokach, L. , & Shapira, B. (2010). Iterative voting underuncertainty for group recommender systems. In Paper presented at the Proceed-

ings of the fourth ACM conference on recommender systems (pp. 265–268) . aamani-Dery, L. , Golan, I. , Kalech, M. , & Rokach, L. (2015). Preference elicitation

for group decisions using the borda voting rule. Group Decision and Negotiation ,1–19 .

aamani-Dery, L. , Kalech, M. , Rokach, L. , & Shapira, B. (2014). Preference elicita-tion for narrowing the recommended list for groups. In Paper presented at the

Proceedings of the 8th ACM conference on recommender systems (pp. 333–

336) . ’connor, M. , Cosley, D. , Konstan, J. A. , & Riedl, J. (2002). PolyLens: a recommender

system for groups of users. In Paper presented at the ECSCW 2001 (pp. 199–218) .ren, J. , Filmus, Y. , & Boutilier, C. (2013). Efficient vote elicitation under candidate

uncertainty. In Paper presented at the Proceedings of the twenty-third interna-tional joint conference on artificial intelligence (pp. 309–316) .

feiffer, T. , Gao, X. A. , Mao, A. , Chen, Y. , & Rand, D. G. (2012). Adaptive polling for

information aggregation. Paper presented at the twenty-sixth AAAI conferenceon artificial intelligence .

ini, M. S. , Rossi, F. , Venable, K. B. , & Walsh, T. (2009). Aggregating partially orderedpreferences. Journal of Logic and Computation, 19 (3), 475–502 .

ossi, F. , Venable, K. B. , & Walsh, T. (2011). A short introduction to preferences: be-tween artificial intelligence and social choice. Synthesis Lectures on Artificial In-

telligence and Machine Learning, 5 (4), 1–102 . enot, C. , Kostadinov, D. , Bouzid, M. , Picault, J. , & Aghasaryan, A. (2011). Evaluation

of group profiling strategies. In Paper presented at the Proceedings of the twen-ty-second international joint conference on artificial intelligence-Volume Three

(pp. 2728–2733) . ervice, T. C. , & Adams, J. A. (2012). Communication complexity of approximat-

ing voting rules. In Paper presented at the Proceedings of the 11th interna-

tional conference on autonomous agents and multiagent systems-Volume 2(pp. 593–602) .

alsh, T. (2007). Uncertainty in preference elicitation and aggregation. In Paper pre-sented at the Proceeding of the national conference on artificial intelligence : 22

(p. 3) . ia, L. , & Conitzer, V. (2008). Determining possible and necessary winners under

common voting rules given partial orders. In Paper presented at the AAAI : 8

(pp. 196–201) . u, Z. , Zhou, X. , Hao, Y. , & Gu, J. (2006). TV program recommendation for multiple

viewers based on user profile merging. User Modeling and User-Adapted Interac-tion, 16 (1), 63–82 .