Learning User Interaction Models for Predicting Web Search Result

PreferenceEugene Agichtein et al.

Microsoft Research

SIGIR ‘06

Objective

• Provide a rich set of features for representing user behavior– Query-text– Browsing– Clickthough

• Aggregate various feature– RankNet

Browsing feature

• Related work

• The amount of reading time could predict– interest level on news articles– rating in recommender system

• The amount of scrolling on a page also have strong relationship with interest

Browsing feature

• How to collect browsing feature?– Obtain the information via opt-in client-side

instrumentation

Browsing feature

• Dwell time

Browsing feature

• Average & Deviation

• Properties of the click event

Clickthrough feature

• 1. Clicked VS. Unclicked– Skip Above (SA)– Skip Next (SN)

• Advantage– Propose preference pair

• Disadvantage– Inconsistency– Noisiness of individual

Clickthrough feature

• 2. Position-biased

Clickthrough feature

Clickthrough feature

Clickthrough feature

• Disadvantage of SA & SN– User may click some irrelevant pages

Clickthrough feature

• Disadvantage of SA & SN– User often click part of relevant pages

Clickthrough feature

• 3. Feature for learning

Feature set

Feature set

Evaluation

• Dataset– Random sample 3500 queries and their top

10 results– Rate on a 6-point scale manually– 75% training, 25% testing– Convert into pairwise judgment– Remove tied pair

Evaluation

• Pairwise judgment

• Input– UrlA, UrlB

• Outpur– Positive: rel(UrlA) > rel(UrlB)

– Negative: rel(UrlA) ≤ rel(UrlB)

• Measurement– Average query precision & recall

Evaluation

1. Current– Original rank from search engine

• 2. Heuristic rule without parameter– SA, SA+N

• 3. Heuristic rule with parameter– CD, CDiff, CD + CDiff

• 4. Supervised learning– RankNet

Evaluation

Evaluation

Evaluation

Conclusion

• Recall is not a important measurement

• Heuristic rule– very low recall and low precision

• Feature set– Browsing features have higher precision

Discussion

• Is user interaction model better than search engine– Small coverage– Only pairwise judgment

• Given the same training data, which one is better, traditional ranking algorithm or user interaction?

• Which feature is more useful?