Full Terms & Conditions of access and use can be found at http://amstat.tandfonline.com/action/journalInformation?journalCode=uasa20 Download by: [Harvard Library] Date: 10 September 2015, At: 18:43 Journal of the American Statistical Association ISSN: 0162-1459 (Print) 1537-274X (Online) Journal homepage: http://amstat.tandfonline.com/loi/uasa20 Bayesian Aggregation of Order-Based Rank Data Ke Deng, Simeng Han, Kate J. Li & Jun S. Liu To cite this article: Ke Deng, Simeng Han, Kate J. Li & Jun S. Liu (2014) Bayesian Aggregation of Order-Based Rank Data, Journal of the American Statistical Association, 109:507, 1023-1039, DOI: 10.1080/01621459.2013.878660 To link to this article: http://dx.doi.org/10.1080/01621459.2013.878660 Accepted online: 14 Jan 2014. Submit your article to this journal Article views: 438 View Crossmark data
Transcript
Full Terms & Conditions of access and use can be found athttp://amstat.tandfonline.com/action/journalInformation?journalCode=uasa20
Download by: [Harvard Library] Date: 10 September 2015, At: 18:43
To cite this article: Ke Deng, Simeng Han, Kate J. Li & Jun S. Liu (2014) Bayesian Aggregation ofOrder-Based Rank Data, Journal of the American Statistical Association, 109:507, 1023-1039,DOI: 10.1080/01621459.2013.878660
To link to this article: http://dx.doi.org/10.1080/01621459.2013.878660
Bayesian Aggregation of Order-Based Rank DataKe DENG, Simeng HAN, Kate J. LI, and Jun S. LIU
Rank aggregation, that is, combining several ranking functions (called base rankers) to get aggregated, usually stronger rankings of a givenset of items, is encountered in many disciplines. Most methods in the literature assume that base rankers of interest are equally reliable. Itis very common in practice, however, that some rankers are more informative and reliable than others. It is desirable to distinguish highquality base rankers from low quality ones and treat them differently. Some methods achieve this by assigning prespecified weights to baserankers. But there are no systematic and principled strategies for designing a proper weighting scheme for a practical problem. In this article,we propose a Bayesian approach, called Bayesian aggregation of rank data (BARD), to overcome this limitation. By attaching a qualityparameter to each base ranker and estimating these parameters along with the aggregation process, BARD measures reliabilities of baserankers in a quantitative way and makes use of this information to improve the aggregated ranking. In addition, we design a method to detecthighly correlated rankers and to account for their information redundancy appropriately. Both simulation studies and real data applicationsshow that BARD significantly outperforms existing methods when equality of base rankers varies greatly.
KEY WORDS: Meta-analysis; Power law distribution; Rank aggregation; Spam detection.
1. INTRODUCTION
Rank aggregation aims to generate a “better” aggregatedrank list (referred to as aggregated ranker) for a set of enti-ties from multiple individual ranking functions (referred to asbase rankers). Early efforts on rank aggregation can be tracedback to studies on social choice theory and political elections inthe eighteenth century (Borda 1781). Since mid-1990s, rank ag-gregation has drawn much attention with the rise of internet andweb search engines. Score-based rank aggregation methods formeta-search (Shaw and Fox 1994; Manmatha, Rath, and Feng2001; Montague and Aslam 2001; Manmatha and Sever 2002),document analysis (Hull, Pedersen, and Schutze 1996; Vogt andCottrel 1999), and similarity search in database (Fagin, Lotem,and Naor 2001), which take score information from individualbase rankers as input to generate an aggregated ranker, formthe first wave of modern rank aggregation studies. However,considering that usually only order information is available inmeta-search, methods that rely only on the order informationfrom base rankers became popular quickly. The first generationof order-based methods construct the aggregated ranking func-tion based on simple statistics of ranked lists from base rankers.For example, Van Erp and Schomaker (2000) and Aslam andMontague (2001) proposed to use a democratic voting proce-dure called Borda count (i.e., the average rank across all baserankers) to generate the aggregated rank; while Fagin, Kumar,and Sivakumar (2003b) suggested the use of median rank. Tostrive for better performance, more complicated methods wereproposed, including Markov-chain-based methods (Dwork et al.2001), fuzzy-logic-based method (Ahmad and Beg 2002), ge-netic algorithm (Beg 2004), and graph-based method (Lam andLeung 2004). As an important special case, the problem of com-bining the top-d lists has been given extra attention in Dwork
Ke Deng, Mathematical Sciences Center, Tsinghua University, Beijing100084, China (E-mail: [email protected]). Simeng Han (E-mail:[email protected]) and Jun S. Liu (E-mail: [email protected]), Depart-ment of Statistics, Harvard University, Cambridge, MA 02138, USA. KateJ. Li, Sawyer Business School, Suffolk University, Boston, MA 02108, USA(E-mail: [email protected]). This research was supported in part by the NSFgrants DMS-0706989, DMS-1007762 and DMS-1208771, and by ShenzhenSpecial Fund for Strategic Emerging Industry grant (No.ZD201111080127A).
Color versions of one or more of the figures in the article can be found onlineat www.tandfonline.com/r/jasa.
et al. (2001) and Fagin, Kumar, and Sivakumar (2003). Freundet al. (2003) proposed a boosting method for rank aggregationwith the guidance of “feedbacks” that provide information re-garding relative preferences of selected pairs of entities.
Randa and Straccia (2003) compared the performance ofscore-based methods and rank-based methods in the context ofmeta-search, and found that Markov-chain-based methods per-formed comparably to score-based methods, but significantlyoutperformed methods based on Borda count. The success ofMarkov-chain-based methods quickly made Dwork et al. (2001)a classic. These methods were later applied to bioinformaticsproblems, and a number of their variations and extensions wereproposed to fit more complicated situations (Sese and Morishita2001; DeConde et al. 2006; Lin and Ding 2009).
In practice, the problem of rank aggregation can become evenmore challenging because of the diverse quality of base rankers.For example, in a meta-search study, some search engines aremore powerful than others; in a meta-analytic bioinformaticstudy, some labs collect and/or analyze data more efficientlythan other labs; and in a competition, some judges are moreexperienced and objective than others. In some extreme cases,some base rankers may be noninformative or even misleading.For example, “paid placement” and “paid inclusion” are verypopular among search engines. These low quality base rankers,referred to as spam rankers, may disturb the rank aggregationprocedure significantly if they are not treated properly. Givingdifferent base rankers different weights, as mentioned in Aslamand Montague (2001) and Lin and Ding (2009), seems to bethe only method available that takes the diverse quality of baserankers into consideration. A clear limitation of this approach,however, is that there are no systematic and principled strategiesfor designing a proper weighting scheme for a practical problem.Supervised rank aggregation may help learn a good weightingscheme, as proposed in Liu et al. (2007), but it needs a good setof training data, which may often be unavailable. To illustratethe main problems and ideas, consider the following simpleexample:
1024 Journal of the American Statistical Association, September 2014
NBA Team Ranking. In July 2012, after the 2011–2012 NBAbasketball season, we sent out a questionnaire to all graduatestudents of the Harvard Statistics Department and a small groupof summer school students who were taking the summer courseSTAT 100 at Harvard, asking them to select the best 8 NBAteams of the 2011–2012 season and rank them top-down basedon his/her own knowledge without checking online informa-tion or consulting others. We also asked each student to classifyhimself/herself into one of the following four groups in the sur-vey: (1) “Avid fans,” who never missed NBA games; (2) “Fans,”who watched NBA games frequently; (3) “Infrequent watch-ers,” who watched NBA games occasionally; and (4) “Not-interested,” who never watched NBA games in the past season.This extra piece of information, which is usually unavailable inmost real problems, will not be used for rank aggregation, butwill be used to validate the quality measure we infer from theranking data. We received 28 responses, amounting to a 47%response rate. The data are displayed in Table 1. We also listin Table 1 six ranking results generated in December 2011 af-ter the preseason games by professional news agencies such asNBA.com, ESPN.com, etc. Assuming that we do not know thequality of each ranker, we wish to combine these rankings intoa better-quality aggregated ranking function, and to also judgeeach ranker’s “quality” based only on these ranking results.
We propose here a Bayesian method to tackle the order-basedrank aggregation problem. By reformulating the original rankaggregation problem into a Bayesian model selection problemand attaching a quality parameter to each base ranker, we canestimate the quality of base rankers jointly with rank aggrega-tion. Compared to existing methods, our method is distinct inthat it uses an explicit probabilistic model, is adaptive to the het-erogeneity of base rankers, and can handle complex situations(such as with correlated rankers of different qualities) more ef-ficiently. The remainder of the article is organized as follows.Section 2 formally defines the rank aggregation problem, andbriefly reviews the existing methods. In Section 3 describes ourBayesian model for rank aggregation, explains intuitions behindthe model, provides details of the corresponding Markov chainMonte Carlo algorithm, and extends the method to handle par-tial rankings and the issue of supervision. Section 4 proposes atool for detecting overly correlated ranker groups and describea hierarchical model to account for information redundancy dueto ranker correlations. Section 5 provides simulation evaluationsof the new Bayesian method, and Section 6 explores a few real-data applications. Section 7 concludes the article with a shortdiscussion.
2. AN OVERVIEW OF EXISTING METHODS
Let U = {1, 2, . . . , n} be the “universe” (set) of n entities ofinterest. An ordered list (or simply, a list) τ with respect to U isa ranking of entities in a subset S ⊆ U , that is, τ = [x1 � x2 �· · · � xd ], where S = {x1, . . . , xd} and “i � j” means that i isranked better than j. Let τ (i) be the position or rank of entityi ∈ τ (a highly ranked element has a low-numbered positionin the list). We call τ a full list if S = U ; and a partial listotherwise. An important special case of partial lists is the top-dlists. For a list τ and a subset T of U, the projection of τ withrespect to T (denoted as τ|T ) is a new list that contains only
entities from T . Note that if τ happens to contain all elementsin T , then τ|T is a full list with respect to T . In the NBA TeamRanking example, U is the set of all 30 NBA teams, and weobserved six full lists P1, . . . , P6 from six professional newsagencies and 28 partial lists S1, . . . , S28 from 28 students.
2.1 Methods Based on Summary Statistics
Many rank aggregation methods are based on simple sum-mary statistics of the m given base rankers. Let {τk(i)}1≤k≤m bethe ranks that entity i receives from the m base rankers. To de-termine the rank of entity i in the aggregated list, the arithmeticmean, geometric mean, or median of {τk(i)}1≤k≤m have all beenproposed. We refer to these three methods as AriM, GeoM, andMedR, respectively. These naive methods are straightforwardand perform reasonably well when rankers in consideration arefull and of similar qualities. But they are easily disturbed byspam rankers and also have difficulties in dealing with partiallists. In the NBA ranking example, we may treat those unrankedteams in a partial list as ranked “# 19.5,” which is at the middlebetween 9 and 30, or an arbitrary number between 9 and 30. Ifthe average approach AriM was used, one may end up rankingLakers and Bulls much higher than they should be.
Dwork et al. (2001) proposed to report the list that minimizesan objective function as the aggregated rank list, that is, let
α = arg minσ∈AU
d(σ ; τ1, . . . , τm),
where AU is the space of all allowable rankings of entities inU, and the objective function d can be either the Spearman’sfootrule distance (Diaconis and Graham 1977)
dF (σ ; τ1, . . . , τm) � 1
m
m∑k=1
F (σ|τk, τk),
where F (σ|τk, τk) = ∑
i∈τk|σ|τk
(i) − τk(i)|, or the Kendall taudistance (Diaconis 1988)
dK (σ ; τ1, . . . , τm) � 1
m
m∑k=1
K(σ|τk, τk),
where K(σ|τk, τk) is the bubble sort distance between σ|τk
and τk
(i.e., the number of swaps that the bubble sort algorithm wouldmake to place σ|τk
in the same order as τk). The aggregationobtained by optimizing the Kendall distance is called Kemenyoptimal aggregation, and the one obtained by optimizing theSpearman’s footrule distance is called footrule optimal aggre-gation. In fact, the idea of generating the aggregated rankingby optimizing the Kendall distance can be traced back to theMallows model in 1950s (Mallows 1957), which is generalizedby Fligner and Verducci (1986), and later by Meila et al. (2007).
Considering that it is computationally expensive to solve theabove optimization problems (the Kemeny optimal aggrega-tion is NP-Hard, and the footrule optimal aggregation needs anexpensive polynomial algorithm), Dwork et al. (2001) also pro-posed a few Markov-chain-based methods as fast alternativesto provide suboptimal solutions. The basic idea behind these
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
Deng et al.: Bayesian Aggregation of Order-Based Rank Data 1025
Tabl
e1.
Pow
erra
nkin
gsof
NB
Ate
ams
for
the
2011
–201
2se
ason
colle
cted
from
six
prof
essi
onal
spor
t-ra
nkin
gw
ebsi
tes
and
asu
rvey
of28
Har
vard
stud
ents
Prof
essi
onal
Avi
dfa
nsFa
nsIn
freq
uent
wat
cher
sN
otin
tere
sted
indi
vidu
als
N.o
.Te
amP
1P
2P
3P
4P
5P
6S
1S
2S
3S
4S
5S
6S
7S
8S
9S
10S
11S
12S
13S
14S
15S
16S
17S
18S
19S
20S
21S
22S
23S
24S
25S
26S
27S
28
1H
eat
12
11
11
11
23
13
–1
21
31
31
41
42
12
–4
––
11
12
2T
hund
er3
32
32
32
23
2–
2–
–4
27
4–
––
2–
–2
––
––
2–
––
–3
Spur
s7
1011
58
76
55
5–
6–
65
4–
–5
–8
63
6–
––
––
––
––
–4
Cel
tics
511
109
95
48
14
25
23
13
43
–4
23
2–
44
–2
––
–2
––
5C
lippe
rs8
56
105
6–
6–
8–
7–
––
––
––
––
––
––
–2
––
––
–4
–6
Lak
ers
67
76
68
37
61
31
12
77
12
12
14
51
31
51
––
84
2–
7Pa
cers
1413
1414
1312
8–
––
––
––
––
––
––
––
––
––
6–
––
58
––
876
ers
1516
1315
1515
––
––
––
–4
––
86
––
––
–5
6–
76
64
33
––
9M
aver
icks
21
32
44
––
77
–8
––
86
65
–8
3–
1–
––
3–
––
4–
5–
10B
ulls
44
44
32
54
86
44
3–
3–
–8
4–
––
––
–3
–3
35
–5
–1
11K
nick
s9
69
87
13–
34
–5
––
––
–2
––
––
––
47
8–
–2
8–
77
612
Gri
zzlie
s10
88
711
10–
––
––
––
––
––
––
––
––
––
––
–8
––
––
–13
Nug
gets
199
513
109
––
––
8–
––
––
––
––
––
7–
––
––
––
––
8–
14M
agic
1112
1711
1411
––
––
––
6–
––
––
–5
––
––
5–
4–
––
66
––
15H
awks
1218
1218
1218
7–
––
––
––
–5
––
––
––
––
87
––
5–
––
–8
16Ja
zz18
2326
2728
19–
––
––
–8
––
––
––
––
––
––
––
––
––
––
317
Tra
ilBla
zers
1314
1512
1614
––
––
––
––
––
––
––
––
––
––
––
––
––
––
18R
ocke
ts21
1516
1617
17–
––
––
–5
76
––
–2
6–
––
3–
5–
5–
3–
––
719
Buc
ks16
1720
1720
16–
––
––
––
––
––
––
3–
––
––
––
––
6–
––
–20
Suns
2022
1921
1921
––
––
––
–8
––
––
77
68
8–
––
––
––
7–
––
21N
ets
1719
2420
2423
––
––
––
––
––
5–
––
––
––
––
––
7–
––
35
22W
arri
ors
2221
2319
2220
––
––
6–
––
––
––
6–
––
––
––
1–
––
––
––
23T
imbe
rwol
ves
2320
2222
2324
––
––
––
––
––
––
––
77
––
––
––
47
––
–4
24H
orne
ts27
2818
2318
25–
––
––
–7
––
––
––
––
––
––
––
––
––
––
–25
Pist
ons
2525
2524
2522
––
––
––
––
––
––
––
––
–8
––
–8
––
2–
––
26K
ings
2924
2126
2126
––
––
––
45
––
––
8–
55
––
––
––
1–
––
––
27W
izar
ds28
2728
2527
27–
––
–7
––
––
––
––
––
––
7–
––
7–
––
–6
–28
Rap
tors
2426
2928
3028
––
––
––
––
–8
––
––
––
––
––
8–
–1
––
––
29C
aval
iers
2629
2729
2629
––
––
––
––
––
–7
––
––
––
–6
––
––
––
––
30B
obca
ts30
3030
3029
30–
––
––
––
––
––
––
––
–6
––
––
––
––
––
–
Rem
ark:
The
30N
BA
team
sar
ear
rang
edin
the
tabl
eba
sed
onth
eir
perf
orm
ance
sin
play
offs
ofth
ese
ason
,tha
tis,
the
top
16te
ams
reac
hed
the
play
offs
,the
top
eigh
ttea
ms
surv
ived
the
first
roun
dof
the
play
offs
,and
soon
.The
six
prof
essi
onal
pow
erra
nkin
gs(P
1,..
.,P
6)
are
dow
nloa
ded
from
FOX
Spor
ts.c
om,E
SPN
.com
,SI.
com
,NB
A.c
om,m
idw
ests
port
sfan
s.co
m,a
ndjs
onlin
e.co
m,r
espe
ctiv
ely.
Mor
ede
tails
abou
tthe
sepr
ofes
sion
alpo
wer
rank
ings
are
give
nin
the
App
endi
x.T
he28
rank
ings
byH
arva
rdst
uden
ts(S
1,..
.,S
28)
are
colle
cted
bya
surv
eyaf
ter
the
2011
–201
2se
ason
was
finis
hed,
inw
hich
each
stud
entw
asas
ked
tose
lect
the
best
eigh
tNB
Ate
ams
inth
e20
11–2
012
seas
onan
dra
nkth
emto
p-do
wn
base
don
his/
her
own
know
ledg
ew
ithou
tche
ckin
gon
line
info
rmat
ion
orco
nsul
ting
othe
rs.T
oco
llect
info
rmat
ion
abou
thow
muc
hth
est
uden
tsfo
llow
edN
BA
gam
esin
the
2011
–201
2se
ason
,we
also
aske
dev
ery
stud
entt
ocl
assi
fyhi
mse
lf/h
erse
lfin
toon
eof
the
follo
win
gfo
urgr
oups
inth
esu
rvey
:(1)
“Avi
dfa
ns”
who
neve
rm
isse
dN
BA
gam
es,(
2)“F
ans”
who
wat
ched
NB
Aga
mes
freq
uent
ly,(
3)“I
nfre
quen
twat
cher
s”w
how
atch
edN
BA
gam
esoc
casi
onal
ly,a
nd(4
)th
e“N
ot-i
nter
este
d”w
hone
ver
wat
ched
NB
Aga
mes
inth
epa
stse
ason
.In
addi
tion,
we
enco
urag
edth
est
uden
tsto
dogu
ess
rand
omly
ifth
eyre
ally
have
noid
eas
abou
tthe
sete
ams.
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
1026 Journal of the American Statistical Association, September 2014
methods is to construct a transition probability matrix P ={pij }i,j∈U based on {τ1, . . . , τm}, where pij is the transition prob-ability from entity i to entity j, and use the stationary distributionof P to generate the aggregated ranked list. More precisely, we let
α = sort(i ∈ U by πi ↓),
where π = (π1, . . . , πn) satisfies πP = π , and symbol “↓”means that the entities are sorted in descending order. Supposethat the current state of the Markov chain is entity i, a fewdifferent transition rules were suggested by Dwork et al. (2001)and Deconde et al. (2006):
MC1: The next state is generated uniformly from the set of allentities that are ranked higher than (or equal to) i by somebase rankers.
MC2: The next state is generated by first picking a base rankerτ at random from all base rankers containing entity i, andthen picking a entity j at random from the set {j ∈ τ : τ (j ) ≤τ (i)}.
MC3: A base ranker τ is chosen at random from all base rankerscontaining entity i and an entity j is chosen at random from allentities ranked by τ , and the next state is set at j if τ (j ) ≤ τ (i)and stay in i otherwise.
MC4: An entity j is generated uniformly from the union ofall entities ranked by all base rankers. If τ (j ) ≤ τ (i) for amajority of base rankers that rank both i and j, then go to j;otherwise, stay in i.
MCT : It is almost identical to MC4, except that the move fromi to j at the last step is not a deterministic procedure basedon the majority vote, but a stochastic procedure in which theprobability for accepting j is proportional to the percentage ofbase rankers that rank j higher than i among all base rankersthat rank both i and j.
2.3 Rank Aggregation of Weighted Lists
Considering that base rankers of interest may not be equallyinformative or reliable in practice, methods based on weightedlists are also proposed. In these methods, each base ranker τk
is assigned a weight wk (0 ≤ wk ≤ 1 and∑
1≤k≤m wk = 1),and the base rankers with larger weights play more importantroles in generating the aggregated list. Aslam and Montague(2001) proposed to generate the aggregated list based on theweighted average of the m lists (known as Borda Fuse), that is,let α = sort(i ∈ U by
∑mk=1 wkτk(i) ↓). Lin and Ding (2009)
extended the objective function of Dework et al. (2001) to aweighted fashion, and generated the aggregated list as follows:
α = arg minσ∈AU
d(σ ; τ1, . . . , τm; w) = arg minα∈AU
m∑k=1
wkd(α|τk, τk),
where d(α|τk, τk) = F (α|τk
, τk) or K(α|τk, τk). The authors
also proposed using cross entropy Monte Carlo (CEMC, seeRubinstein and Kroese 2004) to solve the above optimizationproblem. The optimization method based on Spearman’sfootrule distance is denoted as CEMCF , and that based onKendall distance is denoted as CEMCK .
Although assigning weights to base rankers is a sensible wayof handling the quality difference among them, it can be quite
difficult to design a proper weight specification scheme in prac-tice, especially when little or no prior knowledge on base rankersis available. The supervised rank aggregation (SRA) of Liu et al.(2007) solves this problem at the price of extra training data.In SRA, the true relative ranks of some entities are providedas training data, and the weights {wk}1≤k≤m, which are treatedas parameters instead of prespecified constants in these models,are optimized with the help of the training data as well as theaggregated list σ . A problem of SRA is that no training data areavailable in many applications.
2.4 Rank Aggregation Via Boosting
Another line of using training data to achieve rank aggrega-tion in the literature is the RankBoost method of Freund et al.(2003). Similar to SRA, RankBoost assumes that, besides therank lists {τ1, . . . , τm}, we also have a feedback function ofthe form � : U × U → R, where �(i, j ) > 0 means that en-tity i should be ranked above entity j, �(i, j ) < 0 otherwise,and �(i, j ) = 0 indicates no preference between i and j. Differ-ent from SRA, RankBoost does not assign weights to differentrankings themselves. Instead, RankBoost follows the boostingidea to generate a series of weak rankers from {τ1, . . . , τm}, andconstruct the final ranking by a weighted average of these weakrankers.
As an illustration, we applied AriM, the four MC-basedmethods (MC1, . . ., MC4), and the two CEMC-based methods(CEMCF and CEMCK ) to combine the 34 full/partial rankingslisted in Table 1 without informing any method about qualitiesof the rankers. The new method BARD as described in the nextsection was also applied to the same dataset for a comparisonpurpose. The RankBoost method is not included in the com-parison because feedbacks required by it are not available inthis study. The results are displayed in Table 2 and Figure 8.Figure 8 shows that BARD properly discovered the quality dif-ference among the 34 base rankers. This advantage in turn leadsto a better performance of BARD over other tested methods,which treated all the rankers equally (Table 2). Section 6.2 givesmore details on how BARD was applied to this problem.
3. A BAYESIAN MODEL FOR RANK AGGREGATION
3.1 Assumptions and the Model
Here, we propose a Bayesian approach, called Bayesian Ag-gregation of Rank Data (BARD), to tackle the problem of rankaggregation with rankers of different quality levels. This subsec-tion focuses on the case with full lists; extensions to cases wherepartial lists and training data are involved will be discussed inSection 3.4. The BARD method reformulates the ranking prob-lem as follows. We assume that the set U is composed of twononoverlapping subsets: set UR representing relevant entities(with true signals) and set UB representing noisy backgroundentities. The common goal of each base ranker is to distinguishthe relevant entities from the background ones. By integratingrankings from all base rankers, we attempt to best identify theset of relevant entities.
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
Deng et al.: Bayesian Aggregation of Order-Based Rank Data 1027
Table 2. Performance of different methods on survey data of NBA teams
AriM MC1 MC2 MC3 MC4 CEMC BARD
N.o. Team Mean Rank πi Rank πi Rank πi Rank πi Rank d = F d = K ρi
Remark: (1) for AriM and the MC-based methods, the “Rank” was generated based on “Mean” or “πi” with the Tie Method = random; (2) in CEMC, d = F stands for CEMCF ,and d = K stands for CEMCK , both methods were applied under the default setting where base rankers are equally weighted; (3) BARD was applied under the default setting withhyperparameter p = 16
30 ; (4) the errors in the aggregated rankings are highlighted in bold, and BARD made fewest errors among all tested methods.
Let Ii be the group indicator of entity i ∈ U , where Ii = 1 ifi ∈ UR , and Ii = 0 if i ∈ UB . We make the following assump-tions for base rankers τ1, . . . , τm:
• We have independent rankers. That is, given the indicatorvector I = {Ii}i∈U , the rankings τ1, . . . , τm are mutuallyindependent. A way to detect the violation of this assump-tion and a remedy of the method when the violation occurswill be discussed in Section 4.
• For each base ranker τk , the relative ranks of all backgroundentities τ 0
k � τk|UBare purely random (i.e., uniformly
distributed);• The relative rank of a relevant entity i ∈ UR among the
background entities τ1|0k (i) � τk|{i}∪UB
(i) follows a powerlaw distribution, i.e. P (τ 1|0
k (i) = t) ∝ t−rk , where a largerγk (γk > 0) means that ranker τk can better distinguishrelevant entities from the background ones. Note that byrequiring that γk > 0, we assume that each base ranker τk
is making a good-faith effort for the common goal.• Given τ
1|0k � {τ 1|0
k (i)}i∈UR, the relative ranks of all relevant
entities τ 1k � τk|UR
is purely random (i.e., uniform).
Because the triplet (τ 0k , τ
1|0k , τ 1
k ) gives an equivalent repre-sentation of the information in a full list τk when I is given(the equivalency is illustrated in Figure 1 with a toy exam-ple), the above assumptions lead to the following likelihood
Figure 1. An equivalent representation of a full rank list τk via thetriplet (τ 0
k , τ1|0k , τ 1
k ), where τ 0k and τ 1
k give the internal rankings of thebackground entities and relevant entities respectively, τ
1|0k gives the
relative rank of each relevant entity among background entities.
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
1028 Journal of the American Statistical Association, September 2014
function:
P (τ1, . . . , τm | I, γ )
=m∏
k=1
P (τk | I, γk) =m∏
k=1
P(τ 0k , τ
1|0k , τ 1
k | I, γk
)
=m∏
k=1
P(τ 0k | I
) × P(τ
1|0k | I, γk
) × P(τ 1k | τ
1|0k ; I
), (1)
where P (τ 0k | I ) and P (τ 1
k | τ1|0k ; I ) are uniform distributions on
the corresponding spaces of allowable configurations, and
P(τ
1|0k | I, γk
) =∏i∈UR
P(τ
1|0k (i) | I, γk
)where
P(τ
1|0k (i) = t | I, γk
) ∝ t−γk . (2)
In practice, however, both I and γ are unknown, and it is ourmain goal to estimate them from observations {τ1, . . . , τm}. Let-ting π (I, γ ) denote the prior distribution, we have the followingposterior distribution of (I, γ ):
P (I, γ | τ1, . . . , τm) ∝ P (τ1, . . . , τm | I, γ )π (I, γ ).
Since the marginal probability
ρi � P (Ii = 1 | τ1, . . . , τm) (3)
is a good measurement of the importance of entity i, we generatethe aggregated list as
α = sort(i ∈ U by ρi ↓). (4)
On the other hand, the posterior mean
γ k �∫
γkP (γk | τ1, . . . , τm)dγk (5)
gives the estimation of the quality of base ranker τk .The identifiability of the BARD model can be argued from
the following intuition. If we have a large number m of inde-pendent rankers who generate rankings from the posited model,we will be able to observe a clear gap between the average rankof a relevant entity (across all rankers) and the average rank ofa background entity, which provides us strong evidence to sep-arate relevant entities from background ones. More precisely,using a method of moment we can consistently identify rele-vant entities as m goes to infinity. On the other hand, knowingthe set of relevant entities in turn enables consistent estima-tions of the quality parameters γ as both numbers of relevantand background entities go to infinity. In a practical problemwhere m is of moderate size, having the quality information ofdifferent rankers becomes more useful for efficiently inferringrelevant entities. Thus, inferring the entity indicators vector Iand estimating the quality parameters γ can help each other.
3.2 Motivations and Intuitions Behind the Model
Compared with existing methods, BARD is unique infollowing aspects: (1) it partitions all the entities under rankingconsideration into two groups: the relevant one UR and thebackground one UB , and ignores detailed rankings withineach group (i.e., using the uniform distribution for τ 0
k and τ 1k );
(2) it uses a power-law distribution to model the relative rank ofa relevant entity among all background entities; and (3) it uses anexplicit parameter associated with the power-law distribution to
reflect the quality of a ranker. In this subsection, we explain howthese features might help us better resolve the rank aggregationproblem.
The partition of U into UR and UB is motivated by the obser-vation that behind a ranking problem there is often a partitioningproblem. For example, in the page-ranking problem, conceptu-ally there is a binary answer for each web page whether or notit is truly relevant to a given search task (e.g., a group of keywords). In grant review processes, there is always a binary de-cision for each proposal whether or not it should be funded. Onthe other hand, however, although ranking is often an intermedi-ate step for decision processes that aim to partition entities into“selected” or “unselected” groups, the detailed ranking of eachentity is still important to have in many problems (e.g., proposalrankings). In these cases our partition model is not faithful andsomewhat limited. Fortunately, as shown by our simulationsand real-data applications, under our partition model the in-ferred posterior probability for each entity to be in the “relevantgroup” serves as a good measure to rank the entity.
The power-law model τ1|0k (i) is a convenient approximation
reasonably reflective of reality. In a real problem, the distribu-tion of τ
1|0k (i) depends on many factors and can take different
forms in different problems. But we need to find a computation-ally feasible model for τ
1|0k (i). We reason that it needs to satisfy
the following simple requirement: it should give higher proba-bility to a better rank for a relevant item and be no worse thanassigning it a random (uniform) rank. That is, the probabilityfunction should be a monotone decreasing function. Two obvi-ous choices are exponential or polynomial, of which polynomialis more robust and therefore chosen here. A large range of nu-merical investigations also support the adoption of the powerlaw distribution. For example, we generated each ranker τk asthe order of {Xk,1, . . . , Xk,n}, that is,
τk = sort(i ∈ U by Xk,i ↓),
where Xk,i is generated from two different distributions Fk,0 andFk,1 via the following mechanism:
Xk,i ∼ Fk,0 · I (i ∈ UB) + Fk,1 · I (i ∈ UR), ∀ i ∈ U.
Figure 2 shows that the linear trend in the log-log plot of t versush(t) = P (τ 1|0
k (i) = t | τ 0k ; I, γk) is quite stable across different
specifications of Fk,0 and Fk,1.Third, by modeling τ 1
k and τ 0k with the uniform distribution,
BARD ignores the detailed information on the internal rankingswithin subset UR and UB , and only takes relative rankings be-tween the two subsets into consideration. In other words, wechoose to ignore all information in the data that is ancillaryto the task of separating relevant entities from the backgroundones. This strategy greatly reduces the model complexity andcomputation burden while losses only marginal information inthe data. In some scenarios, we can even argue that internalrankings within the background group are just noise, and thus,should be ignored to stabilize the analysis.
3.3 Details of the Bayesian Computation
Let nI = ∑ni=1 Ii be the number of relevant entities defined by
I. Recall that, for entity i with Ii = 1 (i.e., relevant entity), τ 1|0k (i)
denotes the relative rank of entity i among all the background
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
Deng et al.: Bayesian Aggregation of Order-Based Rank Data 1029
Figure 2. Log-log plots of relative rank τ1|0k (i) = t versus the corresponding probability h(t) = P (τ 1|0
k (i) = t | I, γk) under different scenarios.In each plot, we set |UB | = 1000, thus the range of t is {1, . . . , 1001}. The values of h(t) are calculated via numerical integration.
entities and takes value in ∈ {1, 2, . . . , n − nI + 1}. Accordingto model (2), we have
P(τ
1|0k (i) = ti | I, γk
) = t−γk
i
C(γk, nI ),
and
P(τ
1|0k | I, γk
) =∏i∈UR
P(τ
1|0k (i) | I, γk
), (6)
where the normalizing constant C(γk, nI ) = ∑n−nI +1t=1 t−γk .
Let AURbe the space of all allowable rankings of entities in
UR . Let AUR(τ 1|0
k ) be the configurations of τ 1k that are compatible
with a given τ1|0k . AUR
(τ 1|0k ) is a subset of AUR
due to constraintsintroduced by τ
1|0k . For example, given τ
1|0k = (2, 2, 3) as shown
in Figure 1, τ 1k has only two possible configurations: (1, 2, 3) or
(2, 1, 3), since only the relative position of the first two entitiesE1 and E2 is not fixed given τ
1|0k . In general, we have the fol-
lowing assignment based on the “purely random assumption”:
P(τ 1k = τ | τ
1|0k ; I
) = 1∏n−nI +1t=1 n
1|0τk,t !
· I(τ ∈ AUR
(τ
1|0k
)),
(7)
where n1|0τk,t = ∑
i∈URI (τ 1|0
k (i) = t).Note that the relative rank order of all background entities,
τ 0k , follows the uniform distribution, that is,
P(τ 0k = τ | I
) = 1
(n − nI )!. (8)
Putting (8), (6), and (7) together, we have
P (τk | I, γk)
= P(τ 0k | I
) × P(τ
1|0k | I, γk
) × P(τ 1k | τ
1|0k ; I
)= {(n − nI )! × Aτk,I × (C(γk, nI ))nI × (Bτk,I )γk }−1,
where
Aτk,I �n−nI +1∏
t=1
(n
1|0τk,t !
)and Bτk,I �
∏i∈UR
τ1|0k (i).
Thus, the conditional probability of {τ1, . . . , τm} is
P (τ1, . . . , τm | I, γ ) = [(n − nI )!]−m ×m∏
k=1
{Aτk,I
× (C(γk, nI ))nI × (Bτk,I )γk }−1. (9)
We give I an informative prior
π (I ) ∝ exp− (nI −n·p)2
2σ2 ,
where p is the hyperparameter representing the expected per-centage of relevant entities in U, and set σ 2 as a tunablehyperparameter (whose default value is σ 2 = 1√
m). We let
{γk}1≤k≤m have an independent exponential prior, that is, π (γ ) =∏1≤k≤m f (γk), where f (γk) = λe−λγk , λ is the mean of the ex-
ponential distribution. In BARD, we use λ = 1 as the defaultsetting, and allow the user to specify the value of λ based ontheir own judgment for a practical problem. We also tested usinga uniform prior in hypercube [0, 10]m for γ , which resulted in avery similar performance to the exponential prior.
Given the above prior distributions, we get the joint posteriordistribution of (I, γ ):
P (I, γ | τ1, . . . , τm)
∝ π (I )π (γ )P (τ1, . . . , τm | I, γ )
= π (I )
[(n − nI )!]m·
m∏k=1
f (γk)
Aτk,I × (C(γk, nI ))nI × (Bτk,I )γk,
(10)
which induces the following conditional distributions:
P (γk | τ1, . . . , τm; I, γ[−k])
= P (γk | τk; I )
∝ e−λγk × (C(γk, nI ))−nI × (Bτk,I )−γk , (11)
P (Ii | τ1, . . . , τm; I[−i], γ ) ∼ Bernoulli
(qi(γ )
qi(γ ) + 1
), (12)
where
qi(γ ) = π (I[Ii=1])
π (I[Ii=0])·
m∏k=1
P (τk | I[Ii=1], γk)
P (τk | I[Ii=0], γk).
These distributions enable us to draw samples from P (I, γ |τ1, . . . , τm) via Gibbs sampling. The posterior probabilities,P (Ii | τ1, . . . , τm) and P (γk | τ1, . . . , τm), can be obtained fromthe Monte Carlo samples and used to generate the aggregatedrank list and reliability measures of base rankers. Since the con-ditional distribution shown in (11) is not a standard distribution,we use the random-walk Metropolis algorithm to draw samplesfrom it (see Liu 2001 for a comprehensive review).
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
1030 Journal of the American Statistical Association, September 2014
3.4 Extensions to Partial Lists and SupervisedRank Aggregation
Since a partial list can be viewed as an incomplete version ofa full list, the aggregation of partial lists can be treated as a miss-ing data problem and solved via data augmentation strategies(Tanner and Wong 1987). To be precise, we let {τP
1 , . . . , τPm }
be the m partial lists of interest, and let {τ ∗1 , . . . , τ ∗
m} be theirunobserved underlying full lists. We are interested in drawingsamples from the following target distribution
P(I, γ | τP
1 , . . . , τPm
) ∝ π (I )π (γ )P(τP
1 , . . . , τPm | I, γ
),
which can be achieved via Gibbs sampling based on the follow-ing conditional distributions:
P(τ ∗
1 , . . . , τ ∗m | τP
1 , . . . , τPm ; I, γ
) =m∏
k=1
P(τ ∗k | τP
k ; I, γk
),
P (I, γ |τ ∗1 , . . . , τ ∗
m) ∝ π (I )π (γ )m∏
k=1
P (τ ∗k |I, γk).
Given that the distribution P (I, γ | τ ∗1 , . . . , τ ∗
m) has been an-alyzed in the previous section, we only need to focus onP (τ ∗
1 , . . . , τ ∗m | τP
1 , . . . , τPm ; I, γ ), or more concretely, P (τ ∗
k |τPk ; I, γk) here. Let k be the set of full lists that are compatible
with τPk , we have
P(τ ∗k | τP
k ; I, γk
) ∝ P (τ ∗k | I, γk) · I (τ ∗
k ∈ k).
Again, we can use random walk Metropolis algorithm to drawsamples from this distribution.
BARD can also be applied to the scenario where training dataare available. Let {τ1, . . . , τm} be the m lists (full or partial)of interest, and i1 � i2 � · · · � is be the training information,which gives the true relative rank of s entities {i1, i2, . . . , is} inU. In BARD, a natural way to make use of the training informa-tion is to put constraints on I with respect to i1 � i2 � · · · � is ,that is, if Iit = 1, then Iit ′ = 1 for all t ′ ≤ t . Incorporating thetraining data into the analysis may help BARD better estimatethe quality parameters {γk}1≤k≤m of the m base rankers, andthus, improve the final results.
4. MODEL DIAGNOSTICS AND REMEDIES
4.1 Detecting Violation of the Independence Assumption
Although we will show in Section 5 that BARD is reasonablyrobust to the violation of the “independent rankers” assumption,it is desirable to detect a severe violation of the assumptionand further improve BARD based on this information. Standardcorrelation measures such as the Spearman and the Kendallcorrelations do not work here because any pair of informativerankings are unconditionally correlated since they are supposedto capture the same signal about entity relevancy. This type ofcorrelation is not what we are interested in. Instead, we wish todetect groups of rankings that are “over-correlated” relative totheir quality levels.
Consider all the ranks entity i received from all the rankers,{τ1(i), . . . , τm(i)}. It forms a natural distribution on the rankspace {1, . . . , n}, denoted as Qi . If entity i has a strong pos-itive/negative signal, a significant proportion of the rankerswould give it a high/low rank, so that Qi skews toward the
left/right tail; if entity i belongs to the background, Qi shouldbe close to be uniform. To capture these key features of Qi , wefit Qi with a rescaled Beta distribution:
Qi(t) ∝ dBeta
(t
n + 1; αi, βi
)· I (t ∈ {1, 2, . . . , n}),
where dBeta(x; α, β) is the density of the Beta distribution withparameters (α, β). Assuming that { τ1(i)
n+1 , . . . , τm(i)n+1 } are iid draws
from distribution Beta(αi, βi), we denote the estimated param-eters as (αi , βi), and the fitted distribution as Q(αi , βi) (Qi forshort).
For any pair of base rankers τj1 and τj2 , without loss of gen-erality, we assume that τj1 (i) ≤ τj2 (i). Given the fitted Betadistribution Qi , we use the quantity below to measure excessivecorrelatedness of them at entity i:
V(i)j1j2
�∑
τj1 (i)≤t≤τj2 (i)
Q(t ; αi , βi).
Intuitively, V(i)j1j2
corresponds to the probability that a random
sample from Qi falls into the interval [τj1(i), τj2(i)]. A smallerV
(i)j1j2
means a smaller probability that the two independentrankers agree with each other by chance at entity i, hence astronger evidence of nonindependence. Note that V
(i)j1j2
accountsfor not only the distance between τj1 (i) and τj2 (i), but alsotheir relative probabilities based on Qi . We can estimate thep-value P
(i)j1j2
� P (Vxy < V(i)j1j2
) using Monte Carlo simulation,and summarize the overall evidence for the pair of rankers bythe coordination coefficient:
ζj1j2 � −1
n
n∑i=1
log P(i)j1j2
.
A larger ζj1j2 means that rankers τj1 and τj2 are “over-correlated.”Alternatively, we can use the method of posterior predictivechecking (Rubin 1984) to generate the Bayesian coordinationcoefficient, which will be computationally more demanding.
Under the null hypothesis that the two rankers are indepen-dent, we have by the central limit theorem that ζj1j2 followsN (1, 1
n) approximately, which can be used to set a threshold for
ζj1j2 to claim that τj1 and τj2 are not independent. The proce-dure for discovering correlated rankings can be summarized asfollows:
• For each entity i ∈ U , fit a rescaled Beta distribution Qi
for {τ1(i), . . . , τm(i)};• For each ranker pair τj1 and τj2 , calculate the coordination
coefficient ζj1j2 based on {Qi}i∈U ;• If ζj1j2 is larger than a threshold (e.g., significance level
0.05 with Bonfferoni correction), we say that τj1 and τj2
belong to a “block” of correlated rankers.
4.2 A Hierarchical Model for the CorrelatedBase Rankers
Once the underlying correlation structure among the rankersare detected, we can modify BARD to avoid the negative impactof the correlation. Assume that the correlated base rankers fallinto M blocks {G1, . . . ,GM}, where the rankers within a blockare highly correlated while the rankers from different blocksare conditionally independent given the entity membership I.
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
Deng et al.: Bayesian Aggregation of Order-Based Rank Data 1031
Let G0 be all the other conditionally independent rankers. Tosimplify the problem, we assume that every base ranker providesa complete ranking list in this article. The more general scenarioinvolving partial lists can be solved based on a similar principle.
Let κj be the representative ranker of group Gj , and let γj > 0denote the quality measure of the ranker block Gj . We modifythe BARD model into the following hierarchical form:
P (κj | I, γj ) = P(κ0
j | I)P
(κ
1|0j | I, γj
)P
(κ1
j | I, κ1|0j
),
P (τk | κj , βj ) ∝ exp
{− βj
|Gj | · d(τk, κj )
}.
where βj > 0 measures the average magnitude of correlationbetween κj and base rankers in group Gj , |Gj | is the numberof rankers in group Gj , and d(τk, κj ) is the Spearman’s footruledistance or Kendall tau distance between τk and κj . The jointlikelihood can be written as
P (κ1, . . . , κM ; τ1, . . . , τm | I, γj )
=∏k∈G0
P (τk | I, γk) ·M∏
j=1
⎡⎣P (κj | I, γj )
∏k∈Gj
P (τk | κj )
⎤⎦ .
In words, the model assumes that the base rankers within eachblock Gj are conditionally independent of each other given thecommon ranker κj .
An MCMC sampler for simulating from this distribution can beimplemented based on the following conditional distributions:
P (κj | I, γj , βj , {τk}k∈Gj)
∝ P (κj | I, γj )∏k∈Gj
P (τk | κj )
= P(κ0
j | I)P
(κ
1|0j | I, γj
)P
(κ1
j | I, κ1|0j
)× exp
{− βj
mj
∑k∈Gj
d(τk, κj )
},
P (βj | κj , {τk}k∈Gj)
∝ π (βj ) exp
⎧⎨⎩− βj
mj
∑k∈Gj
d(τk, κj )
⎫⎬⎭ ; and
P (I, γ | κ1, . . . , κM )
∝ π (I )π (γ )M∏
j=1
P (κj | I, γ ).
A random walk Metropolis algorithm can be used to samplefrom P (κj | I, γj , βj , {τk}k∈Gj
). With a noninformative priorfor βj , P (βj | κj , {τk}k∈Gj
) becomes an exponential distribu-tion. Sampling from distribution P (I, γ | κ1, . . . , κM ) can beachieved by the technique developed in Section 3. We useBARDHM to denote this modification of BARD with hierar-chical model.
We note here that a full Bayesian approach that simultane-ously detects correlation block structures and infers parametersin the hierarchical model can be designed rather straightfor-wardly based on the methods described in this section. However,the computational cost for inferring the full hierarchical modelis often too high, and the following two-step approximationstrategy already works very effectively: (1) for each block Gj ,generate its representative ranker κj from {τk}k∈Gj
by a simplemethod (e.g., AriM, MC1, or CEMC); (2) apply ordinary BARDto {κj }mj=1 ∪ {τk}k∈G0 .
5. SIMULATION STUDIES
5.1 Simulation Under the BARD Model
Let U = {1, . . . , n}, of which the first 10% are the relevantentities (i.e., UR = {1, . . . , [n/10]}). We generate the baserankers {τk}1≤k≤m via the following scheme:
τk = sort(i ∈ U by Xk,i ↓), where Xk,i ∼ N (0, 1) · I (i ∈ UB)
+ N (μk, 1) · I (i ∈ UR).
We examine two scenarios: (A) μk = μ for all k, and (B) μk =μ · I (k ≤ m
2 ). In scenario A, the base rankers are equally reli-able; in scenario B, however, only the first 50% base rankers areinformative. The parameter μ controls the signal strength of thedataset (a larger μ means that we have more information to dis-tinguish relevant entities from irrelevant ones). We generate bothfull lists and top-d lists (d = 0.2 · n) for each scenario and testfour cases: full lists from scenario A (denoted as AF ), top-d listsfrom scenario A (denoted as AP ), full lists from scenario B (de-noted as BF ), and top-20 lists from scenario B (denoted as BP ).
We first evaluate the impact of signal strength μ on the per-formance of BARD. Fixing n = 100 and m = 10, we tried fourdifferent values of μ (μ = 0.5, 1.0, 1.5, and 2.0) for each ofthe above four cases. Under each configuration, 1000 indepen-dent datasets were simulated. To each dataset, we applied threenaive methods (AriM, GeoM, MedR), four Markov-chain-basedmethods (MC1, MC2, MC3, MC4), two optimization-basedmethods (CEMCF and CEMCK ), and BARD with λ = 1 un-der three different choices of the hyperparameter p (p1 = 0.05,p2 = 0.10, and p3 = 0.15), respectively. Additionally, we in-clude BARD with the constraint of equal quality, that is,γ1 = · · · = γm (denoted by BARDC), in the comparison. Foreach method, its average coverage rate across the 1000 parallelexperiments under different configurations is calculated to eval-uate the performance. (The coverage rate of an aggregated listis defined as the percentage of true relevant entities covered bythe top-10 entities.)
The results are summarized in Table 3, from which we cansee that: (1) when qualities of base rankers were the same (i.e.,scenario A), BARDC slightly outperformed BARD and achieveda similar performance as CEMC, which is supposed to be “op-timal” in this case; (2) when the quality of base rankers variedgreatly (i.e., scenario B), BARD uniformly outperformed allother methods, and the benefit increased with the increase of thesignal strength μ; (3) both BARDC and BARD were robust tothe choice of the hyperparameter p. Figure 3 displays boxplotsof {γk}k obtained by BARD from the 1000 parallel runs underdifferent configurations, suggesting that BARD was capable of
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
1032 Journal of the American Statistical Association, September 2014
Table 3. Average coverage rates of different rank aggregation methods
Remark: (1) in CEMC, d = F stands for CEMCF , and d = K stands for CEMCK ; (2) BARDC stands for BARD with constraint that γ1 = · · · = γm; (3) for both BARDC and BARD,we tried three values for hyperparameter p, that is, p1 = 0.05, p2 = 0.10, and p3 = 0.15 with hyperparameter λ = 1.
efficiently estimating qualities of base rankers when the signalstrength was reasonably large (e.g., δ ≥ 1.0). We also appliedBARD and BARDC with λ = 2 to each of the simulated datasetand obtained very consistent results, indicting that BARD isrobust to the specification of hyperparameter λ.
We next check the impact of the data size (i.e., num-ber of entities n and the number of rankers m) on the per-formance of BARD. We fixed the signal strength μ = 1.0,and tried two alternative combinations: (m, n) = (10, 200) and(m, n) = (20, 100). The results are summarized into Table 4,from which we can see that most of methods tested were notsensitive to the increase of n, although an increase of m ledto better performances for most methods. More importantly,
BARD performed quite robustly to different choices of n and mcompared to the other methods.
5.2 Robustness of BARD
An important assumption in our model is that the rankers inconsideration work independently, which can often be violatedin real problems. To test how well our method tolerates the viola-tion of this assumption, we simulated 20 rankings {τ1, . . . , τ20}falling into three groups:
G1 = {τ1, τ2, τ3, τ4}, G2 = {τ5, τ6, τ7, τ8}, and
G0 = {τ9, . . . , τ20},
Figure 3. The boxplots of {γk}k estimated by BARD from 1000 parallel runs under different configurations when m = 10 and n = 100 withhyperparameters p = 0.1 and λ = 1.
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
Deng et al.: Bayesian Aggregation of Order-Based Rank Data 1033
Table 4. Impact of data size to the performances of different methods
where rankings in G0 are independently generated, and rank-ings in G1 and G2 have very strong within group correlation.More precisely, we let U = {1, . . . , 100}, let the relevant en-tities be UR = {1, . . . , 10}, and let the background entities becomposed of two subsets: the “neutral” set UB1 = {11, . . . , 90}and the “negative” set UB2 = {91, . . . , 100}. We define δi =I (i ∈ UR) − I (i ∈ UI ), implying that δi = 1 for i ∈ UR , 0 fori ∈ UB1 , and −1 for i ∈ UB2 .
• A ranking τk in G0 was simulated as τk = sort(i ∈U by Xk,i ↓), where Xk,i ∼ N (δi · μk, 1) with μk ≥ 0. Alarger μk means that τk can better separate relevant entitiesfrom background ones, thus a higher quality ranking;
• The rankings in G1 and G2 were generated via two steps:we first generated a common ranking
κ = sort(i ∈ U by Xi ↓) where Xi ∼ N (δi · μ, 1),
and then manipulated κ with random transpositions to gen-erate a group of correlated rankings. Let M(·) denote a
random transposition operation. The aforementioned ma-nipulation can be written as τk = Ms(τ ) where s is num-ber of such operations used. Note that a small s indicates astronger correlation among the rankings.
Fixing μ = μ9 = · · · = μ12 = 0.5, μ13 = · · · = μ16 = 1.0,μ17 = · · · = μ20 = 1.5, we simulated 1000 datasets for each ofthe following three configurations corresponding to s = 20, 60,and 100, respectively. Table 5 shows a typical dataset simulatedwith s = 60, from which we can see that the rankings within G1
or G2 are quite similar to each other for many entities. We ap-plied BARD, BARDHM as well as other methods to each of thesesimulated datasets. The results are summarized in Table 6 andFigure 4. From Table 6, we can see that: (1) BARDHM uniformlyoutperformed all other methods; (2) BARD performed reason-ably well even when correlations among the rankers within G1
and G2 are very strong (i.e., s = 20), and approached the per-formance of BARDHM when the correlation was weaker (i.e.,s = 60 or 100). These results are consistent with the information
Figure 4. Boxplots of {γk}k estimated by BARD from 1000 parallel runs when some base rankers are dependent of each other. The datasetsare simulated from the mechanism described in Section 5.2, where the 20 rankers belongs to three blocks G1 = {τ1, . . . , τ4}, G2 = {τ5, . . . , τ8}and G0 = {τ9, . . . , τ20}.
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
1034 Journal of the American Statistical Association, September 2014
Table 5. A typical simulated dataset for testing the robustness of BARD
Remark: The 100 entities belongs to three subsets UR = {1, 2, . . . , 10}, UB1 = {11, . . . , 90}, and UB2 = {91, . . . , 100}. The entities in UR have strong positive signal, the entities in UB2have strong negative signal, the entities in UB1 do not have strong signal. The 20 rankings fall into three blocks G1 = {τ1, . . . , τ4}, G2 = {τ5, . . . , τ8}, and G0 = {τ9, . . . , τ20}. Rankingsfrom different blocks are generated independently, the rankings in block G0 are generated independently, while the rankings within G1 or G2 come from a common ranking with randommanipulations. The quality of rankings in G1 and G2 is relatively low, while G0 contains rankings at different quality levels.
provided by Figure 4, from which we can see that BARD tendedto overestimate the quality of the rankers in G1 and G2 whenthe correlation within G1 and G2 is very strong (i.e., s = 20).Altogether, these results indicate that BARDHM is efficient indealing with correlated rankers, and BARD is reasonably robustto the model assumptions for rank aggregation.
5.3 Discovery of Highly Correlated Rankers
Here, we test the performance of the proposed coordinationcoefficient for detecting correlation structures among the
rankers. Figure 5 shows the empirical distribution as well as thefitted Beta distribution of Qi for three typical entities from Table5 (entity 1, 11, and 91), suggesting that the Beta-distributionapproximation does effectively capture the key feature of dif-ferent types of entities. We calculated the Spearman correlationmatrix, Kendall correlation matrix and coordination coefficientmatrix for the dataset shown in Table 5, and displayed the resultsin Figure 6 (the second column of subfigure (a)). Similar resultsfor other two datasets simulated under different correlationlevels (s = 20 and 100) are also shown in Figure 6 (the first andthird columns of subfigure (a)). We observe that the proposed
Table 6. BARD is robust to the assumption of “independent rankers”
Naive methods MC-based methods CEMC BARD BARDHM
s AriM GeoM MedR MC1 MC2 MC3 MC4 d = F d = K p1 p2 p3 p1 p2 p3
Figure 5. The natural distribution of {τk(i)}mk=1 and the fitted Beta distribution Qi for three typical entities (i = 1, 11, and 91) in Table 5.
method based on the coordination coefficient worked well in allcases, whereas the correlation coefficients were effective onlywhen the dependence is extremely strong. To better evaluate theperformance of the proposed method, we simulated 100 datasetsfor each of the three correlation levels (s =20, 60, and 100),and calculated the pair-wise discovery rates of the proposedmethod. The results are shown in Figure 6 (b), suggesting thatthe proposed method based on coordination coefficient is indeedeffective to capture the correlation structure of base rankers.
6. REAL DATA APPLICATIONS
6.1 Aggregating Rankings of Cancer-Related Genes
We revisited the interesting work of DeConde et al. (2006) forcombining results from five different microarray-based prostatecancer studies. The first six columns of Table 7 present therankings of the top-25 ranked genes that were found to be up-regulated in prostate tumors compared to normal prostate tissuesfrom five studies (Dhanasekaran et al. 2001; Luo et al. 2001;
Table 7. Bayesian rank aggregation of top-25 genes from five prostate cancer studies
Individual top-25 genes from five prostate cancer studies Top genes reported by BARD Original ranks
Remark: Totally, 89 distinct genes appear in the top-25 lists of the five studies, which are referred to as Luo, Welsh, Dhana, True, and Singh, respectively. And, ρ(k)i stands for vector ρ
obtained from BARD with hyperparameter p = k89 .
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
1036 Journal of the American Statistical Association, September 2014
Figure 6. Performance of the coordination-coefficient based method for simulated data generated from the mechanism described in section5.2, where the 20 rankings fall into three groups G1 = {τ1, τ2, τ3, τ4}, G2 = {τ5, τ6, τ7, τ8}, and G0 = {τ9, . . . , τ20}. The rankings in G0 areindependently generated; the rankings in G1 and G2 have strong within group correlation, since each ranking group are generated from acommon ranking with s random transposition operations. A smaller s means a stronger within group correlation. We simulated 100 datasets fors = 20, 60, and 100 respectively, and applied the proposed method based on the coordination coefficient to each of the 300 simulated datasets.The pair-level discover rates are summarized into figure (b); detailed comparisons with Spearman and Kendall correlation measurements forthree typical datasets are illustrated in figure (a). From the figure, we can see that the proposed method works reasonably well for all cases, whilethe Spearman or Kendall correlation coefficients are effective only when the dependence is extremely strong.
Welsh et al. 2001; Singh et al. 2002; True et al. 2006). These fivestudies relied on different technologies, and their results showthat they are quite different in the genes selected to be includedin the top-25 list. Lin and Ding (2009) analyzed this dataset,found that the gene list in Luo et al. (2001) is the least common
compared to the other four studies, and downgraded its weightin their analysis.
Letting U be the 89 genes appeared in the five top-25 lists,and applying BARD with λ = 1 to this dataset, we obtain con-sistent results under different choices of the hyperparameter p
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
Deng et al.: Bayesian Aggregation of Order-Based Rank Data 1037
01
23
4
γ1 γ2 γ3 γ4 γ5
(a) p = 1089
01
23
4
γ1 γ2 γ3 γ4 γ5
(b) p = 1589
01
23
4
γ1 γ2 γ3 γ4 γ5
(c) p = 2089
Figure 7. Posterior distributions of {γ1, . . . , γ5} obtained from BARD under different hyperparameter p for the dataset of cancer-related genes.
(p = 1089 , 15
89 , and 2089 ). As shown in Table 7, the top genes selected
by BARD under different configurations reflect the consensusof the base rankers, and are robust to the choices of p. As il-lustrated by Figure 7, the gene lists from Welsh et al. (2001)and Dhanasekaran et al. (2001) are relatively reliable, whilethat from Luo et al. (2001) is of a lower quality. However,the Markov-chain-based methods (MC1, MC2, MC3, MC4, andMCT ) gave very poor results when applied to this dataset: inall these methods, the stationary distribution π of the transi-tion matrix P degenerated to a point mass at gene OGT, thatis, πi = 1 if i = OGT and πi = 0 for all the other genes, in-dicating that except OGT, all other genes cannot be effectivelydistinguished.
6.2 Aggregating Rankings of NBA Teams
Ranking sports teams has attracted tremendous attention fromboth sports analysts and academics. Numerous ranking methodshave been proposed for different sports, including NBA, NFL,
MLB, NCAA football, etc. (see Langville and Meyer 2012 fora comprehensive review). The BARD method produces an ag-gregated ranking considering results of any number of rankingmethods, which can be used to provide better predictions ofgame outcomes and to evaluate the effectiveness of differentsports statistics in generating rankings.
As explained in the NBA team ranking example of Section 1and shown in Table 1, we collected six professional rankings and28 amateur rankings for the 30 NBA teams in the 2011–2012season. Defining the 16 teams that entered the 2011–2012 play-offs (i.e., the first 16 teams in Table 1) as “relevant entities,” weexpect BARD to give higher γk’s to both professional rankersand students who had paid more attentions to NBA games.
We applied BARD to the dataset with hyperparameter p = 1630
and λ = 1 to “predict” which teams can make their appearancein the playoffs. The results are summarized into Figure 8. Fromsubfigure (a), we can see that BARD figures out the qualitydifference among different rankers successfully: boxplots ofthe γi’s show a clear decreasing trend with the decrease of the
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33
02
46
Hea
t
Thu
nder
Spu
rs
Cel
tics
Clip
pers
Lake
rs
Pac
ers
76er
s
Mav
eric
ks
Bul
ls
Kni
cks
Griz
zlie
s
Nug
gets
Mag
ic
Haw
ks
Jazz
Trai
lBla
zers
Roc
kets
Buc
ks
Sun
s
Net
s
War
riors
Tim
berw
olve
s
Hor
nets
Pis
tons
Kin
gs
Wiz
ards
Rap
tors
Cav
alie
rs
Bob
cats
0.00.20.40.60.81.0
0 100 200 300 400 500
−25
00−
2350
LogL
ρi
γk
Professional Avid fans Fans Infrequent watchers Not-interested
Teams in the playoffs
Other teams
Figure 8. Results from BARD for aggregating 34 rankings of 30 NBA teams in season 2011–2012.
Dow
nloa
ded
by [
Har
vard
Lib
rary
] at
18:
43 1
0 Se
ptem
ber
2015
1038 Journal of the American Statistical Association, September 2014
basketball knowledge level of the rankers. We also observedsome interesting “outliers” from these boxplots. For example,ranker S5 is an outlier in the group of Avid fans, which preciselyreflects the fact that S5 gave high ranks to Warriors and Wiz-ards, two teams that failed to enter the playoffs. Similarly, thelow quality values estimated for rankers S7 and S8 also reflectcorrectly the fact that both S7 and S8 gave high ranks to multipleweak teams.
Moreover, the aggregated ranking outperformed individualrankings in terms of being closer to the “truth,” even thoughamateur rankings from the students contain considerable amountof noises. The aggregated ranking makes only one mistake:putting Rockets instead of Jazz into playoffs. Among the sixprofessional rankings, however, only P5 achieved the same resultas the aggregated ranking; the other five rankings made twomistakes: P1 missed Nuggets and Jazz, P3 missed Magic andJazz, P2, P4, and P6 missed Hawkes and Jazz.
7. DISCUSSION
In this article, we propose the Bayesian rank aggregation(BARD) method for the rank aggregation problem. By givingeach base ranker a specific quality parameter and estimatingthese parameters using the data, BARD measures the reliabilityof the base rankers in a quantitative way and makes use of thisinformation to improve the aggregated rank list. Compared tothe methods in the literature, BARD works significantly betterwhen the equality of base rankers varies greatly. Both simulationstudies and real data applications demonstrated the usefulnessand superiority of BARD.
BARD assumes that: (1) the entities involved can potentiallybe divided into two subsets: relevant entities UR and back-ground entities UB ; (2) given the group indicators of entitiesI = {Ii}i∈U , the rankers τ1, . . . , τm are conditionally indepen-dent; (3) for each base ranker τk , internal relative rankings ofentities within each subset (i.e., relevant or background) are as-signed randomly, and the rank of a relevant entity among thebackground entities follows a power law distribution. To ap-ply BARD to a practical problem, we need to check whetherthe above assumptions (especially, the first two) hold approx-imately. BARD is reasonably robust if some base rankers aremoderately correlated. However, if correlations among certainrankers are very strong, BARD may report biased results. Sec-tion 4 describes an effective tool for a fast detection of strongranker correlations and a Bayesian hierarchical model to accountfor the correlation structure. Our simulation results showedthat our two-step strategy performed satisfactorily when fac-ing strong ranker correlations.
BARD requires that all base rankers in consideration have acommon objective. That is, at a conceptual level there is a com-mon “true” set of relevant entities UR , and all base rankers aimto rank relevant entities from UR better than background ones.If the data collected in practice do not satisfy this requirement(e.g., the rankings from different base rankers have differentgoals, or are purely based on the opinion of each individualranker), BARD may not be an appropriate tool to use.
In general, BARD is robust to different choices of hyperpa-rameters, such as the expected percentage p of relevant entities inU when p comes from a proper range (e.g., [0.01, 0.2]), and the
prior for ranker quality λ. In some practical problems, choicesof p and λ are obvious. If not, we may need to try differentchoices from proper ranges and check how robust the results arebefore a conclusion can be made.
The framework of BARD supports us to deal with full rank-ings, partial rankings and rankings with ties. It is possible tofurther extend this framework to problems with more compli-cated structures. For example, if some covariates of the entitiesof interest are also observed, which can potentially influencerankings of some base rankers, it is desirable to link these co-variates to the quality parameters of corresponding base rankersto achieve a better performance.
APPENDIX
Detailed information about the professional rankings of NBA teamsused in Section 6.2.
Ahmad, N., and Beg, M. M. S. (2002), “Fuzzy Logic Based Rank AggregationMethods for the World Wide Web,” in Proceedings of the International Con-ference on Artificial Intelligence in Engineering and Technology, Malaysia,pp. 363–368. [1023]
Aslam, J. A., and Montague, M. (2001), “Models for Metasearch,” in Proceed-ings of the 24th Annual International ACM SIGIR Conference on Researchand Development in Information Retrieval, pp. 276–284. [1023,1026]
Beg, M. M. S. (2004), “Parallel Rank Aggregation for the World Wide Web,” inProceedings of the 2004 International Conference on Intelligent Sensing andInformation Processing, Piscataway, NJ: IEEE Press, pp. 385–390. [1023]
Borda, J. C. (1781), “Memoire sur les elections au scrutin,” Histoire del’Academie Royale des Sciences, pp. 657–665. [1023]
DeConde, R. P., Hawley, S., Falcon, S., Clegg, N., Knudsen, B., and Etzioni, R.(2006), “Combining Results of Microarray Experiments: A Rank Aggrega-tion Approach,” Statistical Applications in Genetics and Molecular Biology,5, article 15. [1023,1026,1035]
Dhanasekaran, S. M., Barrette, T. R., Ghosh, D., Shah, R., Varambally, S.,Kurachi, K., Pienta, K. J., Rubin, M. S., and Chinnaiyan, A. M. (2001),“Delineation of Prognostic Biomarkers in Prostate Cancer,” Nature, 412,822–826. [1035,1037]
Diaconis, P. (1988), “Group Representation in Probability and Statistics,” Lec-ture Notes-Monograph Series, 11, Hayward, CA: IMS. [1024]
Deng et al.: Bayesian Aggregation of Order-Based Rank Data 1039
Diaconis, P., and Graham, R. (1977), “Spearman’s Footrule as a Measure ofDisarrary,” Journal of the Royal Statistical Society, Series B, 39, 261–268.[1024]
Dwork, C., Kumar, R., Naor, M., and Sivakumar, D. (2001), “Rank AggregationMethods for the Web,” in Proceedings of the 10th International Conferenceon World Wide Web, pp. 613–622. [1023,1024,1026]
Fagin, R., Kumar, R., and Sivakumar, D. (2003), “Comparing Top k Lists,”SIAM Journal of Discrete Mathematics, 17, 134–160. [1023]
Fagin, R., Lotem, A., and Naor, M. (2001), “Optimal Aggregation Algorithmfor Middleware,” in Proceedings of the 20th ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp. 102–113.[1023]
Fligner, M. A., and Verducci, J. S. (1986), “Distance Based RankingModels,” Journal of the Royal Statistical Society, Series B, 48, 359–369. [1024]
Freund, Y., Iyer, R., Schapire, R. E., and Singer, Y. (2003), “An Efficient Boost-ing Algorithm for Combining Preferences,” Journal of Machine LearningResearch, 4, 933–969. [1023,1026]
Hull, D. A., Pedersen, J. O., and Schutze, H. (1996), “Method Combination forDocument Filtering,” in Proceedings of the 19th Annual International ACMSIGIR Conference on Research and Development in Information Retrieval,pp. 279–287. [1023]
Lam, K. W., and Leung, C. H. (2004), “Rank Aggregation for MetasearchEngines,” in Proceedings of the 13th International Conference on WorldWide Web, pp. 384–385. [1023]
Langville, A. N., and Meyer, C. D. (2012), Who’s #1?: The Science of Ratingand Ranking, Princeton, NJ: Princeton University Press. [1037]
Lin, S. L., and Ding, J. (2009), “Integration of Ranked Lists Via Cross En-tropy Monte Carlo With Applications to mRNA and microRNA Studies,”Biometrics, 65, 9–18. [1023,1026,1036]
Liu, J. S. (2001), “Monte Carlo Strategies in Scientific Computing,” in SpringerSeries in Statistics, eds. P. Bickel, et al., New York: Springer-Verlag. [1029]
Liu, Y., Liu, T., Qin, T., Ma, Z., and Li, H. (2007), “Supervised Rank Aggrega-tion,” in Proceedings of the 16th International Conference on World WideWeb, pp. 481–490. [1023,1026]
Luo, J., Duggan, D. J., Chen, Y., Sauvageot, J., Ewing, M., Bittner, M. L.,Trent, J. M., and Isaacs, W. B. (2001), “Human Prostate Cancer and BenignProstatic Hyperplasia: Molecular Dissection by Gene Expression Profiling,”Cancer Research, 61, 4683–4688. [1035,1037]
Mallows, C. L. (1957), “Non-Null Ranking Models,” Biometrika, 44, 114–130.[1024]
Manmatha, R., Rath, T., and Feng, F. (2001), “Modeling Score Distributionsfor Combining the Outputs of Search Engines,” in Proceedings of the 24thAnnual International ACM SIGIR Conference on Research and Developmentin Information Retrieval, pp. 267–275. [1023]
Manmatha, R., and Sever, H. (2002), “A Formal Approach to Score Normaliza-tion for Meta-search,” in Proceedings of the 2nd International Conferenceon Human Language Technology Research, pp. 98–103. [1023]
Meila, M., Phadnis, K., Patterson, A., and Bilmes, J. (2007), “Consensus Rank-ing Under the Exponential Model,” in Proceedings of the 23rd AnnualConference on Uncertainty in Artificial Intelligence (UAI). [1024]
Montague, M., and Aslam, J. A. (2001), “Relevance Score Normalization forMeta-search,” in Proceedings of the 10th Conference on Information andKnowledge Management, pp. 427–433. [1023]
Randa, M. E., and Straccia, U. (2003), “Web Metasearch: Rank vs. Score BasedRank Aggregation Methods,” in Proceedings of the 2003 ACM Symposiumon Applied Computing, pp. 841–846. [1023]
Rubin, D. B. (1984), “Bayesianly Justifiable and Relevant Frequency Calcula-tions for the Applied Statistician,” The Annals of Statistics, 12, 1151–1172.[1030]
Rubinstein, R. Y., and Kroese, D. P. (2004), The Cross-Entropy Method. AUniÞed Approach to Combinatorial Optimization, Monte-Carlo Simulationand Machine Learning, New York: Springer. [1026]
Sese, J., and Morishita, S. (2001), “Rank Aggregation Method for BiologicalDatabases,” Genome Informatics, 12, 506–507. [1023]
Shaw, J. A., and Fox, E. A. (1994), “Combination of Multiple Searches,” inProceedings of the 2nd Text Retrieval Conference, pp. 243–252. [1023]
Singh, D., Febbo, P. G., Ross, K., Jackson, D. G., Manola, J., Ladd, C., Tamayo,P., Renshaw, A. A., D′, Amico, A. V., Richie, J. P., Lander, E. S., Loda, M.,Kantoff, P. W., Golub, T. R., and Sellers, W. R. (2002), “Gene ExpressionCorrelates of Clinical Prostate Cancer Behavior,” Cancer Cell, 1, 203–209.[1036]
Tanner, M. A., and Wong, W. H. (1987), “The Calculation of Posterior Distri-butions by Data Augmentation,” Journal of the American Statistical Asso-ciation, 82, 528–540. [1030]
True, L., Coleman, I., Hawley, S., Huang, A., Gifford, D., Coleman, R., Beer,T., Gelman, E., Datta, M., Mostaghel, E., Knudsen, B., Lange, P., Vessella,R., Lin, D., Hood, L., and Nelson, P. (2006), “A Molecular Correlate to theGleason Grading System for Prostate Adenocarcinoma,” Proceedings of theNational Academy of Sciences of the USA, 103, 10, 991–10,996. [1036]
Van Erp, M., and Schomaker, L. (2000), “Variants of the Borda Count Method forCombining Ranked Classifier Hypotheses,” in Proceedings of the 7th Inter-national Workshop on Frontiers in Handwriting Recognition, pp. 443–452.[1023]
Vogt, C., and Cottrel, G. W. (1999), “Fusion Via a Linear Combination ofScores,” Information Retrieval, 3, 151–173. [1023]
Welsh, J. B., Sapinoso, L. M., Su, A. I., Kern, S. G., Wang-Rodriguez, J.,Moskaluk, C. A., Frierson, H. F. Jr, and Hampton, G. M. (2001), “Analysis ofGene Expression Identifies Candidate Markers and Pharmacological Targetsin Prostate Cancer,” Cancer Research, 61, 5974–5978. [1036]
![arXiv:1601.05242v2 [math.CA] 24 Jan 2016 · arXiv:1601.05242v2 [math.CA] 24 Jan 2016 Littlewood-Paley Characterizations of Anisotropic Hardy-Lorentz Spaces JunLiu,DachunYang∗ andWenYuan](https://static.documents.pub/doc/80x56/5f0d0b2c7e708231d438660c/arxiv160105242v2-mathca-24-jan-2016-arxiv160105242v2-mathca-24-jan-2016.jpg)
