IRP5: Social Media Data Acquisition

Presented by: Arjen P. de Vries

20/10/2008 2

PROBLEM

• Each of us can think of many research questions related to the Petamedia objective of integrating the data network, the user network and the physical network...

20/10/2008 3

20/10/2008 4

PROBLEM

... but, today, these three networks are mostly disparate – they do not overlap!

So, how to evaluate the effectiveness of our ideas?!

20/10/2008 5

IRP5 OBJECTIVES

• Develop a common data-set that is large enough to allow meaningful research, and contains video content as well as explicit information generated by a social network of sufficient density

• Organize a PetaMedia ‘data-set yellow pages’irp5petamedia.pbworks.com

Functions also as crawling code repository!

20/10/2008 6

20/10/2008 7

DATASET REQUIREMENTS

• Availability of video data• Creative Commons (CC) licenced data only• Data sources with developer-friendly APIs

only

• Availability of social data• Users creating the data should be organized

in a social network, and provide feedback about their preferences in relation to the data (comments, ratings, ...)

20/10/2008 8

CANDIDATE DATA SOURCES

• Blip.tv• high quality, 25-60% CC, poor social data

• Revver.com• Medium quality, 100% CC, poor social data

• Flickr.com• 200K CC,• No API for access to video content

20/10/2008 9

DECISIONS

1) Join Blip.tv and revver.com with Del.icio.us, digg and Twitter to get richer social data

2) Crawl links to videos, as well as the social networks of users creating those links, up to 4 levels of social network but only 2 levels of metadata (bookmarks/posts/profiles)

3) Crawl Flickr irrespective of missing API

Comments,User info,Friend info

Video datablip.tv

revver.com

Social ‘mention’ in•Digg•Del.icio.us•Twitter

20/10/2008 10

FLICKR

• Outline:• Video data acquired through scraping the mobile flickr

site (m.flickr.com)• At most 90 seconds each• Only mp4 (no flv)

• Metadata acquired through API• Typical download rate: 1 video with metadata per

minute• Approach:

• Query for travel-related tags; 10 videos per tag• Leads to 211 videos from 162 uploaders;• Leads to 4143 videos in total from these uploaders• Leads to 17598 videos from their 32K contacts

20/10/2008 11

BLIP+REVVER

• Most popular videos:• Blip-10,000 and Revver-10,000 data sets

• Social mentions of additional videos:• 175 blip.tv and 45 revver.com downloadable

videos mentioned at Del.icio.us (out of 5GB of social data, reached by starting with ‘joshua’ – its founder – and recursively following network fan links)

• 1250 blip.tv and 9198 revver.com video clips digg-ed (by 3602 unique users)

• ~850 blip.tv links posted by Twitter users per week

20/10/2008 12

PRELIMINARY RESULTS

• Del.icio.us (now delicious.com) is better queried over the TU Berlin DAI-Labor lab collection (bookmarks from 2003-2007)• 14K distinct links to blip.tv (from 22K

bookmarks)• 4K distinct links to revver (from 10K

bookmarks)

• Reason: API truncates results to only 100 per item

20/10/2008 13

PRELIMINARY RESULTS

• Twitter is better ‘crawled’ through Topsy, a search engine over Tweets• 27K links to blip (from 42K indexed by Topsy)• 300 links to revver (from ~1200 indexed by

Topsy)

• Reason: API usage limited to #queries per IP address, but, more importantly, API access only to msgs at most 7 days old• BTW: Topsy has been queried using ~2100

‘popular’ `travel-related’ tags, to circumvent 500 results per query limitation

20/10/2008 14

NEXT

• Crawl actual social network data and tweets corresponding to the Twitter V2 and Del.icio.us V2 links, by re-using QMUL and EPFL code

20/10/2008 15

TO BE DONE

• Analyse data and methods• Useful?• Complete?

• Create data repository• Legal check for public sharing• Twitter data• Flickr data html-scraped

• Advertise data sets (SIGIR-Forum?)

20/10/2008 16

TEAM

• TUD:• Pavel Serdyukov (coordination, Del.icio.us V2, Twitter V2)• Stevan Rudinac (blip.tv, revver.com)• Ronald Poppe (blip.tv, revver.com)• Maarten Clements (blip.tv, revver.com)• Arjen P. de Vries (coordination)

• TUB:• Sebastian Schmiedeke (flickr.com)

• EPFL:• Ivan Ivanov (Twitter V1)

• UEP:• David Chudan (Digg)• Tomas Kliegr (Digg)

• QMUL:• Naeem Ramzan (Del.icio.us V1)• Muhammad Akram (Del.icio.us V1)