Date post: | 24-May-2015 |
Category: |
Technology |
Upload: | terrierteam |
View: | 457 times |
Download: | 0 times |
News Article Ranking:Leveraging the Wisdom of Bloggers
Richard McCreadie, Craig Macdonald & Iadh Ounis
Introduction
• Editorial News:• Every day newspaper editors select
articles for placement within their newspapers.
• This can be seen as a ranking problem.
• Rank articles by readership interest
We investigate how such a ranking can be approximated using evidence from the blogosphere
NewspaperEditor
FrontPage
Page2
. . .
Introduction The News Article Ranking Problem The Votes Approach Evaluating Votes Temporal Promotion News Article Representation Conclusions
Talk Outline
News Article Ranking
Problem Definition:• Rank news articles by their inherent
importance.• Given a day of interest dQ we wish to
score each news article a by its predicted importance, score(a,dQ) using evidence from the blogosphere.
News ArticleRanker
=29
=23
=14
=13
=4
=4
ImportanceScores
Day dQ
Idea:• The more blog posts about an article the more important the subject
must be.• Score by blog post volume
ApproachTwo Stages:
1. Score each news article a for all days d based on related blog post volume for day d.
News articles are represented by their headlines
2. Given a query day dQ rank A based on the score for each news article on day dQ, i.e. score(a, dQ)
-> a voting process
The Votes Approach
Votes Approach : Stage 1
Ranking of days for a
For eachnews article a
blog postranking
Votes
2) Select the top 1000 blog posts
for a
Stage 1: Score days for each news story
Days3) Each post votes for a day
votes = 1
2
1
3
4
votes = 2
votes = 0
votes = 2
votes = 1
2
1
3
4
votes = 2
votes = 0
votes = 2
4) Rank days by votes received
Voting Model : Count* Craig Macdonald PhD thesis 2009
Terrier
1) Use its representation
(headline) as a query
Votes Approach : Stage 2
QueryDay 2
Ranking of Articles
Stage 2: Rank news articles for day dQ
News article a1
Stage 1
votes = 1
2 votes = 2
votes = 0
votes = 24
1
3
News article a2
News article a3
votes = 3
4 votes = 6
votes = 1
votes = 23
1
2
votes = 5
1 votes = 9
votes = 0
votes = 73
2
4
2 votes = 2
votes = 12
votes = 52
votes = 3
4 votes = 6
votes = 23
1
1 votes = 9
votes = 0
votes = 73
4
votes = 1
votes = 0
votes = 24
1
3
News article a2
News article a3
Introduction The News Article Ranking Problem The Votes Approach Evaluating Votes Temporal Promotion News Article Representation Conclusions
Talk Outline
Hypothesis:• The volume of relevant blog posts published on a news article is a strong indicator
of that articles importance (from an editors perspective).
Research Questions:• Can the number of related blog posts to a news article published on day dQ provide
a comparative ranking to that which an editor might make?
Evaluating Votes
Setup :• TREC 2009 Blog track top news stories identification task• 100k news headlines from the New York Times to represent articles
• E.g. ‘In a Decisive Victory, Obama Reshapes the Electoral Map’
• Uses blog posts from the Blogs08 blog post corpus (28 million posts)• Judgments for 50 days of interest (dQ’s)
• E.g. 2008-05-22 : headline1 headline34 headline35 headline38
Evaluation:• Mean Average Precision (MAP)
Experimental Setup
dQ Important headlines on dQ
Indexing & Retrieval:• Indexed Blogs08 using Terrier (stemming, stopwords)• Secondary index holds blog post -> day relations• Retrieve 1000 blog posts for headlines.
• DPH (DFR)• BM25
Baselines:• Random ranking : average over 10 runs• Inlinks : hyperlink evidence• TREC 2009 best systems
Experimental Setup
Votes Performance
Results:
Rando
mIn
links
uogT
rTStim
es
KLEClus
Prior
IlpsT
SExp
BM25
+Vot
es
DPH+Vot
es0
0.020.040.060.080.1
0.120.140.160.180.2
MAP
Hyperlink evidence is of less value than textual
evidence
Better performance than TREC 2009 best
systems BM25<DPH (DFR)Votes + extras
TREC 2009 Best Systems Votes Approach
Conclusions:• Blog post volume is a decent indicator of editorial importance• Can be effectively leveraged to rank news articles by their importance• However, still room for improvement (0.17 map)
Votes Performance
How can we improve Votes performance?
Introduction The News Article Ranking Problem The Votes Approach Evaluating Votes Temporal Promotion News Article Representation Conclusions
Talk Outline
Idea• Re-score for each news article using evidence from days before and after dQ.
Intuition• Important stories will be discussed before or after the event
E.g. Run up to an election
Temporal Promotion
1st 2nd 3rd 4th 5th 6th0
1
2
3
4
5
News Article aNews Article b
Days
NumVotes
dQBoth articles receive the same score for dQ under Votes
Hypothesis:• An article which is highly blogged about either before or after dQ should be scored
more highly than one which is not.
Approach:• Promote articles which were highly blogged before or after dQ
• Two Techniques• NDayBoost• GaussBoost
Temporal Promotion
1st 2nd 3rd 4th 5th 6th0
1
2
3
4
5
News Article aNews Article b
Approach• Linearly combines the scores for day dQ with the n days before or after dQ.
NDayBoost
Days
NumVotes
dQ
N = -2
Score=11
Score=6
Idea:• Evidence will weaken as the distance from dQ increases• NDayBoost might over-estimate the importance of days distant from dQ
Approach:• Linearly combine scores as with NDayBoost, but weight each day by its distance
from dQ using a Gaussian curve.
GaussBoost
Distance of days ∆d
Weight
GaussBoost
0 1 2 3 4 5 6 7 8 90
0.2
0.4
0.6
0.8
1
w=0.5w=1w=1.5w=2w=3
∆d
Weight
Weighting• The weight for each article is calculated as : • ∆d is the distance (in days) from dQ • w is the width of the gaussian curve
• Controls the score decay as ∆d increases
1st 2nd 3rd 4th 5th 6th0
1
2
3
4
5
News Article aNews Article b
GaussBoost
Days
NumVotes
dQ
N = -2
Score=11
ScoreGaussBoost(A,4) = (1*4)+(0.79*4)+(0.18*3)
= 7.700
ScoreGaussBoost(B,4) = (1*4)+(0.79*1)+(0.18*1) = 4.970
Score=6
Score=7.700
Score=4.970
Example:• n = -2, w = 1• Weights downward the scores for each day dependent on w.
Hypothesis:• An article which is highly blogged about either before or after dQ is more likely to
be important than one which is not.
Research Questions:• Can the promotion of articles which are highly blogged about before or after dQ
improve article ranking performance?• Does the quality of evidence decrease as distance from dQ increases?• Is historical or future (before or after dQ) blog post evidence more useful?
Research Questions
10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 100.1
0.11
0.12
0.13
0.14
0.15
0.16
0.17
0.18
0.19
0.2
NDayBoostDPH+Votes
NDayBoost PerformanceFuture blog postings does provide useful evidence
Baseline DPH+Votes
Historical evidence is not useful for NDayBoost
n value (days)
MAP
-10
-9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 100.14
0.15
0.16
0.17
0.18
0.19
0.2
GaussBoostDPH+Votes
GaussBoost PerformanceFuture blog postings provide stronger evidence than historical postings
Baseline DPH+Votes
w value (not days!)
MAP
Historical blog postings are useful for days close to dQ
• Conclusions• Both historical and future evidence is useful to improve Votes ranking performance
• Can use this evidence to generate a better ranking for editors if the data is available
• Future evidence is more powerful than historical evidence• Not too useful if we want to rank in real-time though
• NDayBoost is only effective for future evidence• GaussBoost is effective for both future and historical evidence
• The most effective of the techniques• Does not over emphasise evidence from days distant from dQ
Temporal Promotion
Introduction The News Article Ranking Problem The Votes Approach Evaluating Votes Temporal Promotion News Article Representation Conclusions
Talk Outline
Can we improve upon the news article representation?
Issue:• News articles are represented with headlines
• e.g. ‘In a Decisive Victory, Obama Reshapes the Electoral Map
• Headlines are a sparse representation of an article• Many headlines are not `news-worthy’
• Editors don’t even consider these
• e.g. paid death notices
Approach:• Enrich the headlines using related terms extracted from blog posts and Wikipedia.• Prune headlines less likely to be news-worthy
Improving the Article Representation
News Article Enrichment
Idea:• Improve the news-article representation
(headline)• Add related terms (counter sparsity)
Approach:1. Select retrieve top 3 blog posts from:
• Blogs08 (query expansion , K. L. Kwok and M. S. Chan. SIGIR 1998)
• Wikipedia(collection enrichment, F. Diaz and D. Metzler. SIGIR 2006)
using DPH (DFR)2. Expand query with the top 10 terms
identified using Bo1 (G. Amati, Thesis 2003) from those documents.
Terriera
Blogs08/Wikipedia
DPH TopTerms
Bo1
Query expansion/External Query expansion/Collection Enrichment
Article Enrichment:• News headlines while being good quality representations are still ambigious• Collection enrichment helps find the blog posts that are related.
Article Improvement Performance
DPH+Votes Query Expansion Collection Enrichment0.165
0.17
0.175
0.18
0.185
0.19
0.195 Collection enrichment with Wikipedia significantly increases performance
MAP
Article Pruning
Idea:• Editors have lots of latent knowledge to
draw upon• Try simulating this within the system• Prune away articles that an editor would
not even consider
Non-stories:• Remove news articles which follow
editorially defined patterns
Noisy headlines:• Remove misleading dates• Remove uppercase category terms
• Paid Notice• Corrections for the Record• Comments of the Week• Inside the Times• Best Sellers• The Week Ahead• Movie Review
Patterns List: New York Times
• Arts Briefly• The Listings• Dance Review• Whats on Today• Critics Choice• Book of the Times• Music Review
E.g. ‘Inside the Times, November 6, 2008’
E.g. ‘N.F.L. ROUNDUP; Giants Shut Down Tyree for Season; Raiders Cut Hall’
Article Pruning:• Removing non-news-worthy articles makes the ranking of articles easier.
Article Pruning Performance
DPH+Votes Patterns Dates UpperCase All Heuristics0.16
0.1650.17
0.1750.18
0.1850.19
0.1950.2
0.205Patterns significantly increase performance over Votes alone
Dates and Uppercase further increase performance when combined.
MAP
Additive Results?
Technique MAP
DPH+Votes 0.1742
+ GaussBoost 0.1907
+ All Heuristics 0.1996
+ Collection Enrichment 0.1899
All Techniques Combined
0.2210
Idea:• Combine
• Temporal promotion (GaussBoost)• Headline pruning (All Heuristics)• Headline enrichment (Collection
Enrichment)
Results:• Significant increase in performance
over• DPH+Votes• DPH+Votes + Single techniques
Votes:• The volume of blog posts about a news story is a useful measure for the importance from an editorial
perspective• Can be used to automatically rank news stories for a newspaper editor
• The Voting model provides strong baseline ranking performace
Temporal Promotion:• Can be beneficial to look at blog post volume either before or after the day of interest• More useful to look at tomorrows blog posts than yesterdays blog posts• Evidence diminishes as we look further from the day of interest, evidence should be weighted
accordingly
Article representation Improvements• Editors hold much in the way of latent knowledge that we need to simulate
• i.e. they can disregard whole classes of articles as not being news-worthy• By pruning away such articles apriori, ranking performance is improved• Headlines are sparse representations of news articles
• Enrichment with terms from Wikipedia can help find more representative blog posts
Conclusions
TREC 2010:• Blog track top stories identification task is running again in 2010• Focus on real-time ranking of news (no future evidence)• Uses a larger news article collection from Reuters
Future Work
Questions?