Personalized Ranking Model Adaptation for

Web SearchHongning Wang1, Xiaodong He2, Ming-Wei Chang2,

Yang Song2, Ryen W. White2 and Wei Chu3

1Department of Computer ScienceUniversity of Illinois at Urbana-Champaign

Urbana IL, 61801 USAwang296@illinois.edu

2Microsoft Research, Redmond WA, 98007 USA3Microsoft Bing, Bellevue WA, 98004 USA

{yangsong,minchang,xiaohe,ryenw,wechu}@microsoft.com

SIGIR 2013 @ Dublin Ireland 2

Searcher’s information needs are diverse • Exploring user’s search preferences

SIGIR 2013 @ Dublin Ireland 3

Personalization for web search• Exploring user’s search preferences

SIGIR 2013 @ Dublin Ireland 4

Existing methods for personalization • Extracting user-centric features [Teevan et al. SIGIR’05]

• Location, gender, click history• Require large volume of user history

• Memory-based personalization [White and Drucker WWW’07, Shen et al. SIGIR’05] • Learn direct association between query and URLs• Limited coverage, poor generalization

SIGIR 2013 @ Dublin Ireland 5

Personalization for web search• Major considerations• Accuracy

• Maximize the search utility for each single user• Efficiency

• Executable on the scale of all the search engine users• Adapt to the user’s result preferences quickly

SIGIR 2013 @ Dublin Ireland 6

Personalized Ranking Model Adaptation• Adapting the global ranking model for each individual user

SIGIR 2013 @ Dublin Ireland 7

Personalized Ranking Model Adaptation• Adjusting the generic ranking model’s parameters with respect to

each individual user’s ranking preferences

SIGIR 2013 @ Dublin Ireland 8

Linear Regression Based Model Adaptation• Adapting global ranking model for each individual user

Lose function from any linear learning-to-rank algorithm, e.g., RankNet, LambdaRank, RankSVM

Complexity of adaptation

SIGIR 2013 @ Dublin Ireland 9

Instantiation example• Adapting RankSVM [Joachims KDD’02]

reducing mis-ordered pairs

Margin rescaling

Non-linear kernels

SIGIR 2013 @ Dublin Ireland 10

Ranking feature grouping I• Grouping features by name - Name• Exploring informative naming scheme

• BM25_Body, BM25_Title• Clustering by manually crafted patterns

PageRank BM25_Title BM25_Body BM25_Anchor tfidf_title

<qn,dj> 1.0 1.3 0.7 0.2 0.9

<qn,dj> 0.8 0.2 0.3 0.1 0.1

<qm,dk> 0.2 0.7 0.6 0.2 0.5

Group 1 Group 2 Group 3

SIGIR 2013 @ Dublin Ireland 11

Ranking feature grouping II• Co-clustering of documents and features – SVD [Dhillon KDD’01]

• SVD on document-feature matrix• k-Means clustering to group features

PageRank BM25_Title BM25_Body BM25_Anchor tfidf_title

<qn,dj> 1.0 1.3 0.7 0.2 0.9

<qn,dj> 0.8 0.2 0.3 0.1 0.1

<qm,dk> 0.2 0.7 0.6 0.2 0.5

SVD + k-Means

SIGIR 2013 @ Dublin Ireland 12

Ranking feature grouping III• Clustering features by importance - Cross• Estimate linear ranking model on different splits of data• k-Means clustering by feature weights in different splits

PageRank BM25_Title BM25_Body BM25_Anchor tfidf_title

model1 0.20 1.23 0.37 0.32 -0.19

model2 0.78 0.25 -0.32 0.19 0.21

model3 0.14 0.37 0.16 0.22 0.15

k-Means

SIGIR 2013 @ Dublin Ireland 13

Discussions• A general framework for ranking model adaptation• Model-based adaptation v.s. {instance, feature}-based adaptation• Within the same optimization complexity as the original ranking model• Adaptation sharing across features to reduce the requirement of adaptation

data

SIGIR 2013 @ Dublin Ireland 14

Experimental Results• Dataset• Bing.com query log: May 27, 2012 – May 31, 2012• Manual relevance annotation

• 5-grade relevance score• 1830 ranking features

• BM25, PageRank, tf*idf and etc.

SIGIR 2013 @ Dublin Ireland 15

Comparison of adaptation performance• Baselines• Tar-RankSVM

• No adaptation, user’s own data only• RA-RankSVM [Geng et al. TKDE’12]

• Model-based: global model as regularization• TransRank [Chen et al. ICDMW'08]

• Instance-based: reweight annotated queries for adaptation• IW-RankSVM [Gao et al. SIGIR’10]

• Instance-based: reweight user’s click data for adaptation• CLRank [Chen et al. Information Retrieval’10]

• Feature-based: construct new feature representation for adaptation

Applicable in per-user basis adaptation

Only applicable in aggregated adaptation

SIGIR 2013 @ Dublin Ireland 16

Adaptation accuracy I• Per-user basis adaptation

SIGIR 2013 @ Dublin Ireland 17

Adaptation accuracy II• Aggregated adaptation

SIGIR 2013 @ Dublin Ireland 18

Improvement analysis I• Query-level improvement• Against global model

SIGIR 2013 @ Dublin Ireland 19

Improvement analysis II• User-level improvement• Against global model

SIGIR 2013 @ Dublin Ireland 20

Adaptation efficiency I• Batch mode

SIGIR 2013 @ Dublin Ireland 21

Adaptation efficiency II• Online mode

SIGIR 2013 @ Dublin Ireland 22

Conclusions• Efficient ranking model adaption framework for personalized search• Linear transformation for model-based adaptation• Transformation sharing within a group-wise manner

• Future work• Joint estimation of feature grouping and model transformation• Incorporate user-specific features and profiles• Extend to non-linear models

SIGIR 2013 @ Dublin Ireland 23

References1. White, Ryen W., and Steven M. Drucker. "Investigating behavioral variability in web search." Proceedings of the 16th international

conference on World Wide Web. ACM, 2007.

2. Shen, Xuehua, Bin Tan, and ChengXiang Zhai. "Context-sensitive information retrieval using implicit feedback." Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 2005.

3. Teevan, Jaime, Susan T. Dumais, and Eric Horvitz. "Personalizing search via automated analysis of interests and activities." Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 2005.

4. Burges, Chris, et al. "Learning to rank using gradient descent." Proceedings of the 22nd international conference on Machine learning. ACM, 2005.

5. Burges, Chris, Robert Rango and Quoc Viet Le. "Learning to rank with nonsmooth cost functions."Proceedings of the Advances in Neural Information Processing Systems 19 (2007): 193-200.

6. Joachims, Thorsten. "Optimizing search engines using clickthrough data."Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2002.

7. Dhillon, Inderjit S. "Co-clustering documents and words using bipartite spectral graph partitioning." Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2001.

8. Geng, Bo, et al. "Ranking model adaptation for domain-specific search."Knowledge and Data Engineering, IEEE Transactions on 24.4 (2012): 745-758.

9. Chen, Depin, et al. "Transrank: A novel algorithm for transfer of rank learning."Data Mining Workshops, 2008. ICDMW'08. IEEE International Conference on. IEEE, 2008.

10. Gao, Wei, et al. "Learning to rank only using training data from related domain."Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. ACM, 2010.

11. Chen, Depin, et al. "Knowledge transfer for cross domain learning to rank."Information Retrieval 13.3 (2010): 236-253.

SIGIR 2013 @ Dublin Ireland 24

Thank you!Q&A

SIGIR 2013 @ Dublin Ireland 25

Notations• Query collection• from user • for each query

• is a V-dimensional vector of ranking features for a retrieved document• is the corresponding relevance label

• Ranking model• • Focusing on linear ranking models

•

SIGIR 2013 @ Dublin Ireland 26

Instantiation I• Adapting RankNet [Burges et al. ICML’05] & LambdaRank [Burges etal. NIPS’07] • Objective function

• Regularization

SIGIR 2013 @ Dublin Ireland 27

Instantiation I• Adapting RankNet & LambdaRank• Derived gradients

Group-wise updating

SIGIR 2013 @ Dublin Ireland 28

Analysis of feature grouping• Effectiveness of different grouping method• Baseline: random grouping and no grouping