University of Washington

Abstract

Crowdsourced Evaluation for Reranked Twitter Search

Garima Tiwari

Chair of the Supervisory Committee:Professor Dr Ankur Teredesai

Computing and Software Systems

Influence and information diffusion in social networks is often explained by the ’pref-

erential attachment’ growth model which possesses power law degree distributions. In re-

cent years, disseminating information in online social networks through short messages has

gained widespread popularity as evidenced by rapid growth of networks such as Twitter1,

and Facebook2. With the exponential growth of these online social networks social influ-

ence can for the first time be measured over a large population. Computing this influence

of actors in such networks has tremendous utility for various applications such as real-time

search, advertising, marketing, recommendations, expert location, and node classification

to name just a few. Inspite of well established power law models that explain the growth

of such networks, the problem of identifying influential actors is significantly challenging

because both structural as well behavioral properties of the nodes (actors) play a crucial

role in determining influence. Several algorithms have been recently proposed that utilize

the global network structure, as well as exhaustive behavioral attributes of actors. Firstly,

access to global network structures and detailed behavioral attributes is non-trivial to ob-

tain due to limits imposed by the popular platforms. Moreover, there is a lack of robust

evaluation strategy to determine which of the algorithms is better than others in computing

1http://www.twitter.com

2http://www.facebook.com

such influence.

In this paper we first propose a series of novel, efficiently computable influence measures that

only utilize local network properties thereby alleviating the need to have access to the global

network structures. Next we propose measures that combine the behavioral attributes with

local structural properties. We then describe the first ever publicly available ground-truth

data-collection tool we designed to obtain a crowdsourced relative influence/rank data for a

set of twitterers on a variety of topics. We also describe how we compute the baseline inter-

enduser discord between the influence ranks we obtained. Needless to state, we make this

dataset publicly available for future research efforts. Lastly, we describe an exhaustive se-

ries of experiments we conducted to compare and contrast the influence ranks generated by

allthe existing influence computation algorithms. We conclusively demonstrate that there

is no one single algorithm that currently correctly outperforms other algorithms and accu-

rately reflects the enduser influence ranks; but several algorithms do come close to ideal.

Surprisingly, our results indicate that local measures of influence augmented by behavioral

attributes of twitterers are nearly as accurate as any existing exhaustive global measures.

Since influence in microblogs is likely to vary over time, we also provide empirical data

on the variation of influence and discuss the temporal robustness for our proposed local

influence measures.

TABLE OF CONTENTS

Page

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

Chapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 2: Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Chapter 3: Influence measurement Strategies . . . . . . . . . . . . . . . . . . . . . 9

Chapter 4: Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Chapter 5: Facebook Application and Evaluation . . . . . . . . . . . . . . . . . . 19

Chapter 6: Experimental Results and Analysis . . . . . . . . . . . . . . . . . . . . 24

Chapter 7: Time Varying Influence . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Chapter 8: Coclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . 38

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

i

LIST OF FIGURES

Figure Number Page

5.1 Screen Shot of Facebook App Home Page . . . . . . . . . . . . . . . . . . . . 20

5.2 Screen Shot of Facebook App Rerank Tweets Page . . . . . . . . . . . . . . . 21

5.3 Screen Shot of Facebook App Result Page . . . . . . . . . . . . . . . . . . . 23

7.1 how the Tweet rank score of authors of 5 tweet about query ’kinect’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

7.2 how the Tweet rank score of authors of 5 tweet about query ’boxee box’changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . 33

7.3 how the Tweet rank score of authors of 5 tweet about query ’google tv’changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . 33

7.4 how the Tweet rank score of authors of 5 tweet about query ’nissan leaf’changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . 33

7.5 how the Tweet rank score of authors of 5 tweet about query ’ipad’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

7.6 how the Flur rank score of authors of 5 tweet about query ’windows Phone7 launch’ changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . 34

7.7 how the Flur rank score of authors of 5 tweet about query ’SXSW’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

7.8 how the Flur rank score of authors of 5 tweet about query ’halloween’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

7.9 how the Tweet rank score of authors of 5 tweet about query ’mid term elec-tion’ changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . . . 35

7.10 how the Flur rank score of authors of 5 tweet about query ’thanksgiving’changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . 35

7.11 how the Flur rank score of authors of 5 tweet about query ’Microsoft’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

7.12 how the Tweet rank score of authors of 5 tweet about query ’asus’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

ii

7.13 how the Tweet rank score of authors of 5 tweet about query ’zynga’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7.14 how the Flur rank score of authors of 5 tweet about query ’Verizon’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7.15 how the Flur rank score of authors of 5 tweet about query ’Costco’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7.16 how the Flur rank score of authors of 5 tweet about query ’revenue’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7.17 how the tweet rank score of authors of 5 tweet about query ’iran’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

7.18 how the tweet rank score of authors of 5 tweet about query ’facebook’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

7.19 how the Flur rank score of authors of 5 tweet about query ’flu’ changes overa span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

iii

ACKNOWLEDGMENTS

I wish to express my sincere appreciation to University of Washington, where I have had

the opportunity to do my graduate studies and successfully complete this thesis work. I

would like to thank Dr Ankur Teredesai who motivated me to take up this research, and

whose support made it all possible. I would also like to thank Dr Jie Sheng who helped me

through my thesis work.

iv

DEDICATION

to my dear husband, Prashant and my loving daughter, Shravya

v

1

Chapter 1

INTRODUCTION

Sharing short messages through online social networks is an important component of the

Real Time Web that is becoming increasingly popular. Online social network applications

that provide this service to their users, such as Twitter1, and Facebook2 are used by people

all over the world on a daily basis. Recent statistics indicate that Twitter currently has

around 200 million users with more than 1 Billion tweets (messages of up to 140 characters)

being generated per week. Though there is little consensus among social scientists over the

reason why people use these services, it is generally accepted that the ability to express

opinions quickly and freely, and the ability to effectively reach a large audience is the main

draw. From an information consumption perspective, obtaining current trends and the

latest news in real time from a multitude of sources with diverse viewpoints is the main

attraction for the readers of such microblogs [9]. Twitter search, for instance, only provides

keyword matching based search results (i.e. it checks if the tweet contains the search query

or not), and presents them ranked in reverse chronological order. There is no guarantee that

the most interesting tweet appears on top, especially given that thousands of new tweets are

being written every hour. Filtering the Real Time Web to find the most interesting tweet

within a short timeframe is an important challenge. One way to address this challenge is

to first find influential twitterers and then re-rank the tweets based on the influence score.

Computing influence in online social network is of great utility for other domains as well.

For example, influence scores can be used in applications like real time search, advertising,

marketing,recommendation systems etc. If suppose, author a1 has higher influence score

over author a2, then in the real time search result we can show the articles by a1 above

1http://www.twitter.com

2http://www.facebook.com

2

the articles by a2, since author 1 is likely more authoritative, and so information shared by

her is likely more useful. Similarly, if any specific author a1 has higher influence score for

a given topic, say ’iPad’, it implies, he is interested in that topic and so it is beneficial to

show him the advertisement related to apple products rather than any other less relevant

advertisement. Moreover, this model for behavioral targeting can be extended to her social

network who are more likely to listen to ’iPad’ related ads due to their proximity to an

’iPad’ influencer.

Social influence can be computed using many factors including the strength of ties be-

tween actors in the network, the geodesic distance between users, temporal effects, spatial

proximity, and network specific characteristics of the individual actors within the network.

As one can already observe, designing an algorithm for social influence in a network of twit-

terers can involve studying and integrating many such factors into influence measures. Some

of these factors are purely structural while others are behavioral at a node level. Moreover,

structural factors can be global; requiring the entire connected component of the graph, or

local; requiring structural information only few (one or two) hops away from the node under

consideration). Similarly, behavioral factors can be node specific (number of posts, topics

of interest, etc) or collaborative between actors (retweets, mentions, etc).

While the utility of social networks such as Facebook and Twitter depends significantly

on how effective they can be at disseminating information in a trustworthy manner, not

all nodes in such social networks exhibit equal amount of influence. In fact it is now

widely believed that such networks exhibit power law degree distributions [2]. These degree

distributions of many real world social networks which obey a power law are of the form

f(d) ∝ dα (1.1)

with the exponent α > 0, and f(d) being the fraction of nodes with degree d. Such power-law

relations as well as many more have been reported in [1], [22], [7], [10], [13].

Intuitively, power-law-like distributions for degrees state that there exist many low degree

nodes, whereas only a few high degree nodes in real graphs. How do some nodes that start

of as low-degree nodes within a social network gain different properties and exhibit different

3

affinities with other actors and eventually become high degree nodes? Moreover, why are

structural properties not sufficient in identifying influential nodes within modern day online

social networks? Why do many low degree nodes prefer not to listen to actors with high

degrees but rather listen to very some high-degree and some low-degree actors making

influence a combination of structural and behavioral attributes ? Who and why are some

actors more influential than others? Identifying authority figures in online social networks

has drawn considerable attention in recent years.

Recently, several algorithms have been recently proposed that utilize the global network

structure, as well as exhaustive behavioral attributes of actors. Firstly, access to global

network structures and detailed behavioral attributes is non-trivial to obtain due to limits

imposed by the popular platforms. Moreover, there is a lack of robust evaluation strategy

to determine which of the algorithms is better than others in computing such influence.

Weng et al. [19] proposed an extension of page rank algorithm to measure the influence of

twitterers.Pal et al. [15] extracted some detailed metrics and computed them for each po-

tential authoritative twitterers. In another interesting paper Yamaguchi at al. [20] analyzed

User-Tweet graph to compute users’ comparative rankings. Fernandez et al. [8] used more

than 35 variables on twitter and Facebook to compute overall online influence of a user. The

measures used are complex and often require storing and traversing the entire twitter graph

iteratively. In this paper we first propose a series of novel, efficiently computable influence

measures that only utilize local (1 or 2 hop) network properties thereby alleviating the need

to have access to the global network structure and next we propose measures that combine

the behavioral attributes with these local structural properties.

These approaches use their own empirical evaluation strategies and compare results with

degree centrality type measures. Weng et al. [19] did the comparison with related algo-

rithms like indegree, page rank and topic specific page rank. Pal et al. [15] analyzed topic

specific influential twitterers and claim that their algorithm ranks the director of a toy

story 3 higher while rejecting celebrities who tweeted about the movie which proves the

correctness of their algorithm. Hence, we believe there is a substantial need to develop a

4

comprehensive evaluation strategy that takes human judgement into account when deter-

mining effectiveness of influence algorithms. Our proposed tool is one of the first attempts

to solve the evaluation problem. This publicly available tool (hosted on Facebook) collects

human judgements as ground-truth data for influence rank by allowing end-users to treat

influence ranking as a reordering game. Once we get the crowdsourced judgement on which

twitterer is relatively more important than others for a given topic/keyword/category we

compute the baseline inter-enduser discord between the relative influence ranks. We then

generate an average ranking for the twitterers and compare and contrast the accuracy of

various algorithms including the ones we designed. We plan to make this human judgement

rank dataset publicly available for future research efforts.

Since influence of authors in microblogs is likely to vary over time since the behavioral

attributes of the authors and the global structure of the social network varies with time. As

far as we know our work in this domain is the first effort to explore this. We explored the

domain of varying influence of twitterers over a span of 6 months. We provide empirical

data on the variation of influence and discuss the temporal robustness for our proposed

local influence measures.

This paper is organized as follows: We first describe our new efficiently computable

influence measures in chapter 3 that only utilize local network properties thereby alleviating

the need to have access to the global network structures. Next we propose measures that

combine the behavioral attributes with local structural properties. We then describe the

influence rank data-collection facebook app we designed in chapter 5. We describe how

to compute the baseline inter-enduser discord between the influence ranks obtained from

the app. In chapter ?? we describe a series of experiments we conducted to compare

and contrast the influence ranks generated by the various existing influence computation

algorithms. We then discuss our results which indicate that local measures of influence

augmented by behavioral attributes of twitterers are nearly as accurate as any existing

exhaustive global measures. Since influence in microblogs is likely to vary over time, we

also provide empirical data on the variation of influence values with time. Chapter ??

5

concludes the discussion and provides some avenues for future research.

6

Chapter 2

RELATED WORK

It is now well established that analyzing network properties is tremendously useful for

various applications such as identifying authorities, search algorithms [5], [6], [11], for dis-

covering the network value of customers for viral marketing [17], and to improve recommen-

dation systems [4],[18].

Finding influential nodes within a social network is a well studied problem and is often

formulated closely with the related problems of expert identification [?], identification of

hubs and authorities [3], and various variations of the Pagerank [14] algorithm. The actual

problem of finding influentials is significantly different since authority or influence on mi-

croblog networks such as Twitter is determined by a variety of parameters which include

a mix of structural properties of the underlying social network as well as the behavioral

attributes of the nodes themselves. Moreover, our efforts are specifically oriented to find

authority within the Twitter network such that the ranking of their tweets can be done most

effectively, though the measures we design can be generalized very easily to fit other asso-

ciated problems as well. Nagmoti et. al. proposed a set of preliminary measures to obtain

an effective re-ranking of twitter search results [12]. Their influence measures, though local,

suffered from significant evaluation drawbacks which we address in this effort. Weng et. al.

reformulated the problem of finding influence in microblogs to the problem of identifying

topic sensitive influential twitterers [19]. They try to exploit the phenomenon of homophily

in twitter. Homophily implies that a twitterer follows a friend because she is interested

in some topics the friend is publishing and the friend follows back because she finds they

share similar topical interest. Their Twitterrank algorithm is basically an extension of the

iterative Pagerank algorithm and measures the influence of users in twitter given a specific

topic. It measures the influence by taking into account both topical similarity between

7

users and and the link structure but is constrained by both the availability of the global

network connectivity data and dependent on the accuracy of the topic-model generated by

the LDA model [?]. Topic modeling within fast moving text streams such as microblogs can

itself be a significantly challenging task [21] and is computationally expensive. Moreover

the influence computation in Weng et.al.’s approach significantly depends on the quality of

the topic model which can be difficult to ground [16].

Other ranking algorithms that utilizes only the underlying structural properties of the

twitter graph are TURank and ObjectRank [20]. Unlike PageRank, ObjectRank takes

account of edge types and node types in order to deal with multiple kinds of edges and

nodes. Pal et. al describe an approach where a list of metrics are extracted and computed

for each potential authority [15] . They divide each tweet by a twitterers into 3 categories

original tweet(OT), conversational tweet(CT) and repeated tweet (RT). OTs are tweets that

are not RT or CT. A CT is directed at another user, denoted by the use of @username token

preceding the text. RT are produced by someone else but the user copies, or forwards, them

in-order to spread it in her network. These metrics are: RT@username, Number of original

tweets, Number of links shared, Self-similarity score that computes how similar is authors

recent tweet w.r.t. to her previous tweets, Number of keyword hashtags used , Number

of conversational tweets, Number of conversational tweets where conversation is initiated

by the author, etc. Next they extract a few features across the tweets of a user on the

topic of interest such as; (a) topical Signal - estimates how much an author is involved with

the topic irrespective of the types of tweets posted by her, (b) signal strength - measures

originality of author’s tweets and it indicates how strong is the twitterers topical signal, (c)

non chat signals - to discount the fact that the author did not start the conversation but

simply replied back out of courtesy, (d) retweet impact - indicates the impact of the content

generated by author or how many times it has been retweeted by others, (e) mention impact

- used to find how much an author is mentioned with regard to a topic of interest, and, (f)

retweet impact - indicates the impact of the content generated by the author. Many such

measures have been employed recently to identify twitterer authority and influence. One

8

such popular authority as a service API is provided by KLOUT.com 1. The scores range

from 1 to 100 with higher scores representing a wider and stronger sphere of influence. Klout

uses over 35 variables on Facebook and Twitter to measure True Reach, Amplification

Probability, and Network Score. In this paper we evaluate and demonstrate how even

KLOUT scores are inadequate indicators of influence. In another interesting paper authors

have given a brief description of the 5 algorithms which can be used to rank twitter users.

These are pagerank [14], HITS, Noderanking ,tunkrank, twitterrank [19].

The lack of conclusive comparison between approaches is one main drawback we address

in our paper. Moreover, we demonstrate that local measures of influence computed based

on one or two hop networks for any given twitterer are as good at determining authority as

any of the global network measures previously suggested in literature.

1www.klout.com

9

Chapter 3

INFLUENCE MEASUREMENT STRATEGIES

Nagmoti et al [?] defines three measures for measuring influence of authors of microblogs

and then propose how to use these measures for ranking microblogs, in combination with

some properties of the microblogs themselves, such as the length of a microblog and whether

the microblog contains a URL.

3.0.1 Ranking Authors of Microblogs

Let A denote the set of all authors and Q the set of all queries. By a ranking measure for

authors, we mean a mapping g : A×Q → R+ that associates with every author-query pair

(a, q) a nonnegative real number g(a, q), called the rank of author a w.r.t. query q. This

implies that the rank of an author can vary with the query topic, in other words that an

author can be considered as more authoratative on some topics than on others.

The first strategy proposed ranks authors based on the number of tweets they have

posted so far in the microblogging system. The underlying idea is that

active publishers might be more valuable as information sources than inactive publishers.

[TweetRank] Let a be an author and q a query, then the TweetRank of a w.r.t.q is defined

as

TR(a, q) = N(a) (3.1)

with N(a) the total number of tweets posted by a so far.

As the right hand side in Formula (3.1) clearly reveals, TweetRank is a query independent

measure, i.e., the TweetRank of an author is the same over all query topics. The second

ranking strategy that proposed is query independent as well. It is purely based on the posi-

10

tion of an author in the social network of the microblogging service. This social network is

a directed graph in which an edge from user u to user v means that u is following v. In this

case, u is called a follower of v, and u will find all posts of v automatically displayed on his

account page. Furthermore, v is called a followee of u. Intuitively, an author is influential if

he has a lot of followers. Indeed, if an author is spreading very useful information, naturally

many people follow him. FollowerRank captures this idea.

[FollowerRank] Let a be an author and q a query, then the FollowerRank of a w.r.t. q is

defined as

FR(a, q) =i(a)

i(a) + o(a)(3.2)

with i(a) being the indegree of a, i.e. the number of followers of a, and o(a) being the

outdegree of a, i.e. the number of users followed by a.

Here, the number of followers of an author is divided by his total in and out degree. This

acts as a damping factor to the FollowerRank of authors who have a unusually high number

of followers just because they are socially overactive, rather than because of the quality of

their tweets. FollowerRank varies between 0 and 1.

3.0.2 Ranking Microblogs

Let T denote the set of all tweets and Q, as before, the set of all queries. Ranking measure

for tweets mean a mapping f : T ×Q → R+ that associates with every tweet-query pair (t, q)

a nonnegative real number f(t, q), called the rank of tweet t w.r.t. query q. Furthermore,

let auth denote the T → A mapping that maps every tweet t to its author auth(t). The

ranking measures for authors proposed above can be used to rank tweets as query search

results using any of the functions defined below.

[Ranking measures for tweets] The ranking measures fTR, fFR, and fRR for tweets are

defined as

11

fTR(t, q) = TR(auth(t), q)

fFR(t, q) = FR(auth(t), q)

fRR(t, q) = RR(auth(t), q)

(3.3)

for all t ∈ T and q ∈ Q.

In addition to the social network based ranking strategies proposed above, two more

factors are considered which may indicate the amount of information shared through tweets,

namely the presence of a URL (http link) in a tweet, and the length of a tweet. The amount

of information contained within a tweet can at times be proportional to the length of the

tweet. This leads to a query dependent ranking strategy for tweets termed LengthRank, a

measure which varies between 0 and 1.

[LengthRank] Let t be a tweet and q a query, then the LengthRank of t w.r.t. q is defined

as

fLR(t, q) =l(t)

maxn∈T (q,k)

l(n)(3.4)

with T (q, k) the set of top-k tweets on query topic q, and l(t) and l(n) the length of tweet

t and n respectively.

The author’s intention behind sharing a URL is mostly to direct his audience towards

some potentially interesting information available somewhere else on the web, so the pres-

ence of a URL can be an indication of informativeness. This leads to a query independent

measure called URLRank.

[URLRank] Let t be a tweet and q a query, then the URLRank of t w.r.t. q is defined as

fUR(t, q) =

c if t contains a URL

0 otherwise(3.5)

with c a positive constant.

12

As a stand-alone ranking strategy, this measure will not be very effective because many

tweets will receive rank 0 (because they do not contain a URL) and the rest will receive

the same rank c, which does not allow for a relative ordering of tweets. Like LengthRank,

URLRank can however be used in a meaningful way in combination with any of the social

network based ranking strategies, leading among others to the ranking measures proposed

in Definition 3.0.2.

[Ranking measures for tweets]

The ranking measures fULR,fFLR and fFLUR for tweets are defined as

fULR(t, q) = fUR(t, q) + fLR(t, q)

fFLR(t, q) = (fFR(t, q) + fLR(t, q))/2

fFLUR(t, q) = fFLR(t, q) + fUR(t, q)

(3.6)

for all t ∈ T and q ∈ Q.

3.0.3 Other Ranking strategies

Apart from the influence based ranking strategies proposed by Nagmoti et al [?], we de-

veloped a couple new and innovative strategies to test against the crowdsourced evaluation

data collected by our FB app described in chapter 5.

The first strategy proposed here, called Combined Influence Score, uses the composition

of following metrics:

[OnSubject metric] Let n be the number of tweets from author a on query q and m be

the number of tweets from author a, then the OnSubject metric of i is defined as

OSM(a) =n

m(3.7)

[Chatty metric] Let n be the number of original tweets of author i, then the chatty

metric of i is defined as

13

CYM(a) =n

maxj(chatty(j))(3.8)

here j ranges over all authors in the social network graph to normalize the result.

[Concise metric] the Concise metric of a is defined as

CM(a, q) =maxtweetlength− avg(a)

maxtweetlength(3.9)

here avg(a) is the average tweet length of all tweets from author a on query q.

[Link Structure metric] Let n be the number of followers of a and m be the number of

friends of author a, then the Link Structure metric of a is defined as

LSM(a, q) =n

n+m(3.10)

[Combined Influence Rank] Let OSM be OnSubject Metric for a, CYM be Chatty

Metric for a, CM be Concise Metric for a and LSM be Link Structure Metric for a, then

the Combined Influence Score of a is defined as

CIS(a, q) = OSM + CYM + CM + LSM (3.11)

Another new strategy proposed in this thesis, called Content Based Influence Score,

which gives a lot of emphasis on the content posted by the authors in order to determine

their influence. In this strategy the tweets of authors are split up into words and then

searched through for a count of how many times each word appears.The words are divided

into 4 buckets : weak, fail, actions, and good

To compute the score of words, the Individual word scoring system is as follows :

• The author is given points for each word used.

14

• A user loses some points for each set of words between a pair of commas that is fewer

than 4 words.

• A user loses some points for each word that ends with ... and that does not start with

htt.

• All of the points for tweets are divided by the number of words to normalize them.

Each word is given a multiplier, where it is -1 for bads, -2.1 for fails, 1.2 for goods, and

1.5 for actions. For each word, if the multiplier is negative, and the frequency is less

than or equal to 0.015 out of all words, 20000*frequency*(0.015 - frequency) . If the

multiplier is negative and the frequency is greater than 0.015, the formula is instead

20000*0.015*(frequency - 0.015) . If the multiplier is positive and the frequency is

between 0 and 0.03, the formula is instead 20000*frequency*(0.03 - frequency). The

word score is then increased by 1 to make it more likely to be positive. If it is still

negative, it is replaced by 0.0000000000000001.

[FinalWordScore] Let FWS be final word score for a word w , then theFinalWordScore

of a word w defined as

FWS(w) = (wordScore)1/5 (3.12)

where wordScore is computed by above mentioned Individual word scoring system.

[Content based Influence Score] Let CBIS be Content based Influence Score for an

author , then it is defined as

CBIS(a, q) = (∑i

(∑w∈W

(FWS(w))) (3.13)

where W is the set of words in a tweet, i ranges from 0 to n n is the total number of tweets

of author a.

Also we studied recently popular authority computation service provided by KLOUT.com 1.

In our thesis work we evaluate how klout’s influence score based ordering of authors is sim-

ilar to our crowdsourced evaluation data. Klout score is the measurement of overall online

1www.klout.com

15

influence of a user. The scores range from 1 to 100 with higher scores representing a wider

and stronger sphere of influence. Klout uses over 35 variables on Facebook and Twitter

to measure True Reach(It is the size of user’s engaged audience.), Amplification Probability

(It is the likelihood that user’s content will be acted upon. How often do user’s messages

generate retweets or spark a conversation?) and Network Score( It is the influence level

of user’s engaged audience. Engagement is measured based on actions such as retweets,

@messages, follows, lists, comments, and likes.).

16

Chapter 4

DATASET

Not all nodes in a social networks exhibit equal amount of influence. Several algorithms

have been recently proposed that utilize the global network structure, as well as exhaustive

behavioural attributes of authors. There is a lack of robust evaluation strategy to determine

which of the algorithms is better than others in computing such influence. Therefore we

decided to build a publicly available ground-truth data-collection tool to obtain a crowd-

sourced relative influence/rank data for a set of twitterers on a variety of topics.

We maintain a large dataset with tweets retrieved for the 20 query words under consid-

eration, 4.2, from twitter api along with the publicly available social network information

about their authors. This information is used to compute the rankings for tweets and their

authors based on our influence measuring ranking strategies as described in Chapter ??.

Out of all the tweets available in the dataset for a query word, we select a set of random 5

tweets. In order to collect the crowdsourced evaluation data, Each set of 5 tweets is shown

to 3 unique end users of our facebook app (described in chapter 5) corresponding to the

query selected. Thus we maintain a set of 5 tweets for each query word along with their

authors and other structural and behavioural attributes. This dataset of 96 authors (Since

4 of the authors were repeated in this dataset) their 96 tweets and their graph information

that is publicly available through twitter API is used for the evaluation purpose of influence

measuring ranking strategies. Also we maintain the ordering of tweets submitted by the 3

end users for each query.Using this we compute the average ordering of tweets by the end

user. To explain this lets take an example. Suppose there are 3 end users A, B, C who

use facebook app and provide their ordering for a set of tweets T1,T2,T3,T4,T5 here T1

represent tweet1 and T2 represent tweet2 and so on. The table ??hows the ordering of

tweets by 3 end users A, B, C.

17

position A’s ordering B’s ordering C’s ordering

1 T3 T3 T3

2 T2 T2 T2

3 T1 T1 T4

4 T5 T4 T1

5 T4 T5 T5

Table 4.1: Ordering of 5 tweets by 3 authors

Influence of authors in microblogs is likely to vary over time since the behavioural at-

tributes of the authors and the global structure of the social network varies with time. In

order to be able to observe the change in the authors’ scores based on various influence

measuring strategies, we also maintain the timestamped information of these 96 authors

under consideration. Thus we maintain twitter graph information from December 2010 to

May 2011 at irregular intervals for the 96 authors under consideration.

In December 2010 when we collected the data of 96 authors,all of them were active and

their data was publicly available but during the span of 6 months of data collection we

observed that some of these 96 authors got suspended from twitter api and some of them

got their account protected and so their social network details were no longer available to

us. Unfortunately for one of the query words, ie. toy story 3, 4 out of the 5 authors under

consideration got suspended/protected. Since in this scenario, for the above mentioned

query, the ordering of authors/tweets was not possible to be maintained at various times

due to unavailability of authors’ social network information to generate influence score/rank

based on our influence measuring ranking strategies therefore we dropped this query for

evaluation purposes.

18

id query word

1 boxee box

2 i pad

3 kinect

4 nissan leaf

5 google tv

6 halloween

7 Thanskgiving

8 SXSW

9 windows phone 7 launch

10 mid term elections

11 verizon

12 costco

13 microsoft

14 asus

15 zynga

16 flu

17 facebook

18 revenue

19 toy story 3

20 iran

Table 4.2: Query Words

19

Chapter 5

FACEBOOK APPLICATION AND EVALUATION

Currently, there is no standard ground truth dataset available to evaluate ranking strate-

gies for Twitter. All the reranking strategies mentioned in chapter 3 seem promising but in

the absence of any ground truth dataset to compare the reranked tweets it was difficult to

concretely say anything about the correctness of these algorithms. Previously Nagmoti et al

[?] had evaluated the performance of these strategies using preference judgement. Later it

was realised that preference judgement is not considered widely accepted way of evaluation.

Therefore we decided to measure the accuracy of the proposed re-ranking strategies using

human assessors for evaluation and collect ground-truth data for aiding future research in

this domain. We therefore developed a Facebook application called ’Twitter Ranking’1

to collect crowd sourced evaluation data. For the facebook app (shown in 5.1) , first we

selected 4 categories of query words as in table 4.2. Each of these categories consist of 5

trending query words. When an end-user selects a query words a set of 5 tweets related to

that query word are shown to the end-user in an arbitrary order 5.2 . This set of 5 tweets

is shown to 3 unique end users of facebook app. Then the user is supposed to drag and

drop these 5 tweets in an order that seems best to him. In other words he/she can place the

most informative tweet on top and least informative tweet at the bottom. After the user

submits his/her ordering the application shows him how much he/she was similar to our

ranking strategy and who else on facebook ranked the tweets in a similar ordering 5.3. This

is done to make the Facebook application more interesting and engaging for our end-users.

This also makes it a contest to see who matches up to whom and leads to people sharing

the application with their friends.

1http// : www.apps.facebook.com/twitter ranking

20

Figure 5.1: Screen Shot of Facebook App Home Page

The authors information and the tweets’ content are used to compute the rankings for

tweets based on our ranking strategies. We make sure that each set of 5 tweets are shown

to three unique end users to get their ordering of tweets. The discord value between the

rankings of any 5 tweets by users and by algorithm is computed using Kendalls tau distance.

Kendall tau distance is a metric that counts the number of pairwise disagreements be-

tween two lists or orderings. The larger the distance the more dissimilar the two lists are.

K tau distance can be defined as total number of discordant pairs.

K-tau distance between 2 lists τ1 and τ2 is given by:

21

Figure 5.2: Screen Shot of Facebook App Rerank Tweets Page

K(τ1, τ2) = |(i, j) : i < j, (τ1(i) < τ1(j) ∧ τ2(i) > τ2(j)) ∨

(τ1(i) > τ1(j) ∧ τ2(i) < τ2(j))|

K(τ1, τ2) =

0 if the two lists are identical

n(n− 1)/2 if the two lists are reverse(5.1)

we have normalized the K-tau by dividing by (n(n-1)/2)*10 (here n = 5) so that 0 indi-

cates identical and 10 indicates maximum disagreement between the two orderings.

22

For each query word a set of 5 tweets is presented to 3 unique facebook app users and

their ordering is collected and saved using our app. Then using this saved data, the average

ordering by the 3 users who ranked a set of same 5 tweets is estimated for each query word.

Next the kendalls tau distance metrics is used to find the discordance between this average

ordering of 5 tweets by the 3 end users of FB app and the ordering of same 5 tweets by our

various reranking strategies. We also compute the inter user discord values to find out the

discordance between the 3 end users who order the same set of 5 tweets for a query word.

We have discussed this inter-user discord and the discord between humans and reranking

strategies in chapter 6.

23

Figure 5.3: Screen Shot of Facebook App Result Page

24

Chapter 6

EXPERIMENTAL RESULTS AND ANALYSIS

For the 4 categories of queries, we computed the discordance between the average or-

dering of 5 tweets by the 3 end users of FB app and the ordering of same 5 tweets by our

various reranking strategies using Kendalls tau distance metric explained in 5. Result are

shown in tables 6.1 to 6.4. We also computed the inter human discord between the 3 FB

app end users who ordered the 5 tweets for each query word to estimate how much the

humans were in agreement when they ordered the tweets using Facebook app. (shown in

the 1st columns of tables 6.1 to 6.4).

The last row in each of the tables 6.1 to 6.4 has average values of the the inter human

discord and discord with various influence measuring strategies explained in 3. Discussing

about the

Analysing the table 6.1, we see for some query words (ex. boxee box) LengthRank,

which is an author independent ranking measure, gives minimum discord with the human

assessors whereas for other query words (ex. google tv) Length rank gives very high discord

value. Also for some query words (ex. ipad ) FLUR rank seems to be closest to human

assessors while for other query words (ex. kinect) Tweet Rank Seems to be doing better

25

query human discord discord discord discord discord discord discord discord

word discord with with with with with with with with

FR LR TR FLUR ULR KR SNA1 SNA2

kinect 1.33 5 3 3 4 3 2 3 5

boxee box 2.66 8 2 7 4 4 6 6 8

google tv 2.66 9 8 9 10 8 9 9 7

nissan leaf 3.33 3 2 6 3 2 6 8 6

i pad 4.66 1 2 3 0 4 1 2 1

Average 2.928 5.2 3.4 5.6 4.2 4.2 4.8 5.6 5.4

Table 6.1: Evaluation of reranking strategies against human assessors on facebook for Prod-uct Category (The inter-user discord of the 3 users who ordered the same set of 5 tweetsfor a query word in column 2).The discord value between average of 3 FB users ordering and the ordering by variousreranking strategies lies between 0 and 10. Here FR: FollowerRank, LR: LengthRank,TR: TweetRank, FLUR:FollowerLengthURLRank, ULR: URLLengthRank, KR: KloutScor-eRank, SNA1: SocialNetworkRank1, SNA2:SocialNetworkRank2

26

query human discord discord discord discord discord discord discord discord

word discord with with with with with with with with

FR LR TR FLUR ULR KR SNA1 SNA2

windows 2.66 4 4 8 4 4 7 4 5

Phone 7

launch

SXSW 3.33 8 1 3 2 2 4 2 7

halloween 3.33 6 3 5 2 2 4 4 8

mid term 6 5 5 4 4 3 4 4 5

election

thanksgiving 6.66 5 4 5 4 3 3 7 6

Average 4.396 5.6 3.4 5.0 3.2 2.8 4.4 4.2 6.2

Table 6.2: Evaluation of reranking strategies against human assessors on facebook for EventCategory (The inter-user discord of the 3 users who ordered the same set of 5 tweets for aquery word in column 2).The discord value between average of 3 FB users ordering and the ordering by variousreranking strategies lies between 0 and 10. Here FR: FollowerRank, LR: LengthRank,TR: TweetRank, FLUR:FollowerLengthURLRank, ULR: URLLengthRank, KR: KloutScor-eRank, SNA1: SocialNetworkRank1, SNA2:SocialNetworkRank2

27

query human discord discord discord discord discord discord discord discord

word discord with with with with with with with with

FR LR TR FLUR ULR KR SNA1 SNA2

microsoft 3.33 7 9 7 5 4 5 9 6

asus 4 4 3 1 3 3 2 2 3

zynga 4 7 3 5 6 3 4 7 8

verizon 4 5 5 5 4 4 5 6 5

costco 6 6 7 6 6 7 7 9 8

Average 4.266 5.8 5.4 4.8 4.8 4.2 4.6 6.6 6.0

Table 6.3: Evaluation of reranking strategies against human assessors on facebook for Com-pany/Organization Category (The inter-user discord of the 3 users who ordered the sameset of 5 tweets for a query word in column 2).The discord value between average of 3 FB users ordering and the ordering by variousreranking strategies lies between 0 and 10. Here FR: FollowerRank, LR: LengthRank,TR: TweetRank, FLUR:FollowerLengthURLRank, ULR: URLLengthRank, KR: KloutScor-eRank, SNA1: SocialNetworkRank1, SNA2:SocialNetworkRank2

28

query human discord discord discord discord discord discord discord discord

word discord with with with with with with with with

FR LR TR FLUR ULR KR SNA1 SNA2

revenue 0.66 5 7 3 3 4 2 6 4

iran 3.33 5 6 4 5 4 4 3 5

facebook 4.66 3 2 4 3 3 5 3 5

flu 4.66 6 4 5 1 1 0 7 5

Average 3.3275 4.75 4.75 4 3 3 2.75 4.75 4.75

Table 6.4: Evaluation of reranking strategies against human assessors on facebook for Mis-cellaneous Category (The inter-user discord of the 3 users who ordered the same set of 5tweets for a query word in column 2).The discord value between average of 3 FB users ordering and the ordering by variousreranking strategies lies between 0 and 10. Here FR: FollowerRank, LR: LengthRank,TR: TweetRank, FLUR:FollowerLengthURLRank, ULR: URLLengthRank, KR: KloutScor-eRank, SNA1: SocialNetworkRank1, SNA2:SocialNetworkRank2

query average human discord

revenue 0.66

kinect 1.33

windows phone 7 launch 2.66

boxee box 2.66

google tv 2.66

Table 6.5: The query words with minimum average human discord : The average humandiscord is average of the discord values of the 3 users who ordered the same set of 5 tweetsfor a query word on FB app)

29

than FLUR rank.

Also The fact that URLLength rank,which is an author independent influence measure,

performs better for some query words (ex. nissan leaf) than other influence measuring

strategies, indicates that that despite the very constrained size restriction on tweets, the

differences in length and presence of a URL still hold useful clues on the relative informa-

tiveness of tweets.

Another interesting indication by the table 6.1 is that when we used the klout score for

ranking the authors,for some queries (ex.ipad),the discord is as low as 1 but the similar

low discord value is also achieved by a pretty simple influence measuring strategy FR. For

the query words where discord values are high when using Klout (ex. Nissan leaf), indi-

cates that the human assessors are not always in agreement with the ordering of authors’

influence by Klout, which uses over 35 variables to compute the influence score. Thus the

analysis of table 6.1 shows that there is no one single algorithm that currently correctly out-

performs other algorithms and accurately reflects the end-user influence ranks; but several

influence measuring algorithms do come close to ideal. Surprisingly, our results indicate

that local measures of influence augmented by behavioral attributes of twitterers are nearly

as accurate as any existing exhaustive global measures.

30

Chapter 7

TIME VARYING INFLUENCE

Influence of authors in microblogs is likely to vary over time since the behaviroul at-

tibutes of the authors and the global structure of the social network varies with time. As

far as we know our work in this domain is the first academic effort to explore this. We

explored the domain of varying influence of twitterers over a span of 6 months. We provide

empirical data on the variation of influence and discuss the temporal robustness for our

proposed local influence measures. To analyse this, as explained in the chapter 4, we col-

lected enough tweets related to each of the 20 query words from twitter api along with their

authors’ information at 3 timestamps. We used our Facebook app to collect the ordering

for a set of 5 tweets for each query by 3 unique human assessors. We selected 5 tweets and

corresponding 5 authors for each query word and thus observed the 96 (since 4 authors were

repeated in the dataset) authors over a span of 6 months ie. from dec 2010 to may 2011.

The authors various statistics changed with time and so their ranks based on our reranking

strategies. This change in authors’ ranks over time is important to decide if time needs to

be taken under consideration while designing the rearnking strategies.

Based on the timestamped data we collected, we have drawn graphs to show how the au-

thors rank scores for various queries change over time.For some query words the some of

the authors under consideration got suspended by Twitter or got their account protected so

their information is not available to public. Due to this only those number of authors who

exist from Dec 2010 to May 2011 are shown in the graph. For the sake of brevity we are only

showing the best performing algorithm’s change over time for each query. In order to find

out the best reranking strategy for the 20 query words, the reranking strategies described

31

in Chapter 3 have been used. The best strategy is the one which has minimum discord1

(see table 6.1) with the average end user ordering.

To explain the time varying influence graphs, lets take figure 7.2, here the comparative

influence of the related 5 authors stays the same over the span of 6 months. In other words,

author 1 who had highest influence score (Highest tweet rank value) in December 2010 stays

on the top throughout since he has the highest influence score till May 2011.Similarly the

author 2 has higher influence score than author 3 throughout the span of 6 months. thus

we can see, during this span the twitterers comparative ranks stays the same and therefore

time does not change the influence of authors in this case.

In another graph,7.3, the author 3 who had the highest score in December 2010 goes

down to 2nd highest score in May 2011.Also author 4 who had the 2nd highest score in

december 2010 goes down to the 3rd highest score in May 2011. Interestingly the author 1

who was at the 3rd spot based on his tweet rank score in December 2010, goes up to the 1st

spot in May 2011. Thus in this case the comparative ranks of influential twitterers changes

over time.

We are presenting the graphs for the 4 categories of queries ie. Products, Events, Com-

pany/organizations and Miscellaneous. The products category has 4 query words namely,

kinect, boxee box, google tv , nissan leaf, i pad. Based on the dataset the minimum discord

for kinect out of the reranking strategies described in 3 was tweet rank and therefore the

time varying tweet rank scores of related 5 authors for this query are shown in 7.1.

32

For the best 5 query words for which the average human discord was minimum as

described in chapter 6(see table 6),in 2 out of the 5 cases the comparative ordering of

authors changed with time.

After observing all the timestamped social network properties of authors and correspond-

ing graph we found that more than 75 percent times the comparative ranking of authors

stays the same and does not change with time. As we have shown in graph 7.2 where the

authors’ hold their comparative rank and it does not change with time. In other words the

author 1 had highest TR score and Author 5 had least tweet rank score in December and

this pattern stays the same till May 2011 and thus the influence score ordering of authors

stays the same across a span of 6 months. Less than 25 percent of times the ordering of

authors based on our reranking strategies change as shown in graph 7.14 where the FLUR

rank score of author 2 was higher than author 1 in December 2010 but in May 2011 The

FLUR rank score of author 1 becomes higher than that of author 2. Since less than 25

percent times there is change in authoritative ordering of authors, we can say there is not

an intense need to add temporal dimension in the reranking strategies.

1if the best strategy is constant performing algorithm over time the second best one is used in graph

33

Figure 7.1: how the Tweet rank score of authors of5 tweet about query ’kinect’ changes over a span of 6months

Figure 7.2: how the Tweet rank score of authors of 5tweet about query ’boxee box’ changes over a span of6 months

Figure 7.3: how the Tweet rank score of authors of 5tweet about query ’google tv’ changes over a span of 6months

Figure 7.4: how the Tweet rank score of authors of 5tweet about query ’nissan leaf’ changes over a span of6 months

34

Figure 7.5: how the Tweet rank score of authors of5 tweet about query ’ipad’ changes over a span of 6months

Figure 7.6: how the Flur rank score of authors of 5tweet about query ’windows Phone 7 launch’ changesover a span of 6 months

Figure 7.7: how the Flur rank score of authors of 5tweet about query ’SXSW’ changes over a span of 6months

Figure 7.8: how the Flur rank score of authors of 5tweet about query ’halloween’ changes over a span of 6months

35

Figure 7.9: how the Tweet rank score of authors of 5tweet about query ’mid term election’ changes over aspan of 6 months

Figure 7.10: how the Flur rank score of authors of 5tweet about query ’thanksgiving’ changes over a spanof 6 months

Figure 7.11: how the Flur rank score of authors of 5tweet about query ’Microsoft’ changes over a span of 6months

Figure 7.12: how the Tweet rank score of authors of5 tweet about query ’asus’ changes over a span of 6months

36

Figure 7.13: how the Tweet rank score of authors of5 tweet about query ’zynga’ changes over a span of 6months

Figure 7.14: how the Flur rank score of authors of 5tweet about query ’Verizon’ changes over a span of 6months

Figure 7.15: how the Flur rank score of authors of 5tweet about query ’Costco’ changes over a span of 6months

Figure 7.16: how the Flur rank score of authors of 5tweet about query ’revenue’ changes over a span of 6months

37

Figure 7.17: how the tweet rank score of authors of5 tweet about query ’iran’ changes over a span of 6months

Figure 7.18: how the tweet rank score of authors of 5tweet about query ’facebook’ changes over a span of 6months

Figure 7.19: how the Flur rank score of authors of 5tweet about query ’flu’ changes over a span of 6 months

38

Chapter 8

COCLUSION AND FUTURE WORK

39

BIBLIOGRAPHY

[1] Aggarwal, C. C. Social Network Data Analytics. Kluwer Academic, 2011.

[2] Barabasi, A., and Albert, R. Emergence of Scaling in Random Networks. InProceedings of the National Academy of Sciences (1999).

[3] Bell, R., Koren, Y., and Volinsky, C. Hubs, authorities, and communities. InACM Computing Surveys (CSUR)Volume 31 Issue 4es, (1999).

[4] Bell, R., Koren, Y., and Volinsky, C. Modeling relationships at multiple scales toimprove accuracy of large recommender systems. In 13th ACM SIGKDD internationalconference on Knowledge discovery and data mining, pages 95104, New York, NY,USA, (2007).

[5] Borodin, A., Roberts, G., Rosenthal, J., and Tsaparas, P. Link analysis rank-ing: algorithms, theory, and experiments. In ACM Trans. Inter. Tech., 5(1):231297(2005).

[6] Chakrabarti, S., Dom, B., Kumar, S., Raghavan, P., Rajagopalan, S.,Tomkins, A., Gibson, D., and Kleinberg, J. Mining the webs link structure.In Computer, 32(8):6067 (1999).

[7] Faloutsos, M., Faloutsos, P., and Faloutsos, C. On powerlaw relationships ofthe internet topology. In SIGCOMM, pages 251262 (1999).

[8] Fernandez, J. klout-for-business-a-sometimes-useful-metric-but-an-incomplete-view-of-customers. http://www.web-strategist.com/blog/2011/02/21/klout-for-business-a-sometimes-useful-metric-but-an-incomplete-view-of-customers/comment-154250645.

[9] Java, A., Song, X., Finin, T., and Tseng, B. Why We Twitter: UnderstandingMicroblogging Usage and Communities. In Proceedings of WebKDD and SNAKDD(2007).

[10] Kleinberg, J., Kumar, R., Raghavan, P., Rajagopalan, S., and Tomkins,A. The Web as a graph: Measurements,models and methods. In Computer Science,1627:117 (1999).

40

[11] Kumar, R., Raghavan, P., Rajagopalan, S., and Tomkins, A. Core algorithmsin the clever system. In ACM Trans. Inter. Tech.,6(2):131152, (2006).

[12] Nagmoti, R., Teredesai, A., and De Cock, M. Ranking Approaches for Mi-croblog Search. In IEEE/WIC/ACM International Conference on Web Intelligenceand Intelligent Agent Technology (2010).

[13] Newman, M. Power laws, pareto distributions and zipfs law. In Contemporary Physics(2005).

[14] Page, L., Brin, S., Motwani, R., and Winograd, T. The PageRank CitationRanking: Bringing Order to the Web. In in Proceedings of the Seventh InternationalWeb Conference(WWW 98) (1998).

[15] Pal, A., and Counts, S. Identifying Topical Authorities in Microblogs. In proceedingsof WSDM 2011 (2011).

[16] Ramage, D., Dumais, S., and Liebling, D. Characterizing micorblogs with topicmodels. In In Proceedings of Fourth International AAAI Conference on Weblogs andSocial Media, pages 130137, (2010).

[17] Richardson, M., and Domingos, P. Mining knowledge-sharing sites for viral mar-keting. In KDD (2002).

[18] Victor, P., Cornelis, C., Cock, M., and Teredesai, A. Trust- and Distrust-Based Recommendations for Controversial Reviews. In Intelligent Systems, IEEE,Volume: 26, Issue: 1 (2011), pp. 48 – 55.

[19] Weng, J., Peng, E., Jiang, J., and He, Q. Twitter Rank: Finding Topic SensitiveInfluencial Twitterers. In Proceedings of the third ACM international conference onWeb search and data mining (2010).

[20] Yamaguchi, Y., Takahashi, T., Amagasa, T., and Kitagawa, H. TUrank:Twitter User Ranking based on User-Tweet Graph Analysis. In Proceedings of Wise2010:11th International Conference (2010).

[21] Yao, L., Mimno, D., and McCallum, A. Efficient Methods for Topic Model Infer-ence on Streaming Document Collections. In Knowledge Discovery and Data Mining(KDD) (2009).

[22] Zhan, Y.and Chakrabarti, D., and Faloutsos, C. R-MAT: A recursive modelfor graph mining. In SIAM Int. Conf. on Data Mining (2004).