Center for the Study of New Media and Societywww.newmediacenter.ruData Retrieval BasicsSergey Chernov

Data Expert Evolution5/24/2013 Sergey Chernov, Information Retrieval Basics1995 1999 2013Everyone is a dataexpert today!

Outline Open Data Online and Social Media Data Sources Multimedia Archives Hands-on AssignmentSergey Chernov, Information Retrieval Basics

Outline Open Data Online and Social Media Data Sources Multimedia Archives Hands-on AssignmentSergey Chernov, Information Retrieval Basics

Open Data Open data is the idea that certain data should befreely available to everyone to use and republish asthey wish, without restrictions from copyright,patents or other mechanisms of control.5/24/2013 Sergey Chernov, Information Retrieval Basics

Open Gov Data Several national governments have created web sites todistribute a portion of the data they collect.5/24/2013 Sergey Chernov, Information Retrieval Basics

Data.gov US Open Gov Data5/24/2013 Sergey Chernov, Information Retrieval Basics

Opengovdata.ru5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://www.facebook.com/pages/OpenGovDataru/139336139422921

Open Gov: Open Data5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://../

Roskomnadzor (Federal Service for Supervision ofCommunications, Information Technology and Mass Media)5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://www.rsoc.ru

Roskomnadzor Open Data5/24/2013 Sergey Chernov, Information Retrieval Basics

Roskomnadzor Open Data (cont)5/24/2013 Sergey Chernov, Information Retrieval Basics

Russian Ministry of Education and Science5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://./%D0%BE%D1%82%D0%BA%D1%80%D1%8B%D1%82%D1%8B%D0%B5-%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D0%B5

Rosstat (Russian Federal StateStatistics Service)5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://www.gks.ru

Rosstats Interactive Database5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://cbsd.gks.ru/

Single Cross-Departament InfomationStatistical System ()5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://www.fedstat.ru/indicators/start.do

Moscow Open Data Portal5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://data.mos.ru/

Example: Moscow Museums5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://data.mos.ru/datasets/426_muzei/map/

Moskomarchitecture Geoportal5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://egip.mka.mos.ru/egip/egip.nsf/va_GeoDataCatalogByCat?OpenView

Moscow Open Budget5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://budget.mos.ru/gp_xml

Openpolice.ru5/24/2013 Sergey Chernov, Information Retrieval Basics

Apps4Russia Annual contest ofopendata-based mobileapplications Last winner projectSotskartochka (MySocial Card)5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://www.mysocialcard.ru/http://apps4russia.ru/

GIS-LAB5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://gis-lab.info/start.html

Outline Open Data Online and Social Media Data Sources Multimedia Archives Hands-on AssignmentSergey Chernov, Information Retrieval Basics

Factiva5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://www.dowjones.com/factiva/-28 languages from 200 countries over 35 years- Top news sources include The Wall Street Journal, Dow Jones Newswires, The NewYork Times, The Sydney Morning Herald and Le Monde.

Medialogia5/24/2013 Sergey Chernov, Information Retrieval Basicsmlg.ru

Integrum5/24/2013 Sergey Chernov, Information Retrieval BasicsIntegrum.ru

5/24/2013 Sergey Chernov, Information Retrieval Basicswww.mediacloud.org

Media Cloud Twitter vs LiveJournal5/24/2013 Sergey Chernov, Information Retrieval Basics

Russian Media Cloud5/24/2013 Sergey Chernov, Information Retrieval Basics

Russian Media Cloud (cont)5/24/2013 Sergey Chernov, Information Retrieval Basics

Russian Media Cloud: Association Golos5/24/2013 Sergey Chernov, Information Retrieval BasicsMedia / ClusterElectionsMonitoring andViolationsReportsElect-oralLegis-lationPressureon Golosobserversin regionsODIHRobser-versFine forillegalagitationNTV and"Surkovs Propa-ganda"Shiba-novasnote-bookRoskom-nadzorwarningDDOSattacksTOTALkommersant.ru 51 19 -- 9 2 -- -- -- -- 81golos-org LJ 29 24 7 3 -- -- -- -- -- 63gazeta.ru 22 2 -- 1 9 3 6 1 2 46echo.msk.ru/blog 13 18 1 -- 1 -- -- -- 1 34newsru.com 11 -- -- -- 3 3 1 2 2 22svpressa.ru 9 2 -- -- 1 -- 2 -- 3 17lenta.ru/ 2 -- -- -- 3 1 3 2 4 15www.kavkaz-uzel.ru 12 1 -- -- 1 -- -- -- 1 15aif.ru 11 3 -- -- -- -- -- -- -- 14anticompromat LJ 5 2 -- -- 1 3 2 -- 1 14vz.ru 3 -- -- -- 4 6 1 -- -- 14bfm.ru 2 -- -- -- 4 1 -- 1 4 12www.news2.ru 3 2 -- -- 3 1 1 1 1 12fontanka.ru 1 1 -- -- 3 2 2 1 1 11

VisualizingData withGephi5/24/2013 Sergey Chernov, Information Retrieval Basicsgephi.org

Facebook5/24/2013 Sergey Chernov, Information Retrieval BasicsCities Facebook Userscity_code1 city1 602 106603 2804 605 38206 - 207 5208 3209 2010 382011 640012- 62013 16014 358015 38016 40

Vk.com5/24/2013 Sergey Chernov, Information Retrieval Basics- Server with 12 cores and 40Tb of space- Several weeks of crawling- PostgreSQL database- 120 mln user profilesCities VK Users Facebook Userscity_code1 city1 4312 602 89141 106603 6336 2804 1805 605 24208 38206 - 3548 207 12697 5208 11280 3209 5918 2010 42679 382011 68572 640012 - 21929 62013 1603 16014 27148 358015 2368 380http://habrahabr.ru/post/123856/

Twitter API5/24/2013 Sergey Chernov, Information Retrieval Basics

Yandex Blog Search5/24/2013 Sergey Chernov, Information Retrieval Basics

Topsy5/24/2013 Sergey Chernov, Information Retrieval Basics

SocialBro5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://userguide.socialbro.com/post/40596560247/using-socialbro-to-reach-the-maximum-audience-with-your

Social Bro (cont)5/24/2013 Sergey Chernov, Information Retrieval Basics

NodeXL5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://nodexl.codeplex.com/

Outline Open Data Online and Social Media Data Sources Multimedia Archives Practical AssignmentSergey Chernov, Information Retrieval Basics

Library of Congress5/24/2013 Sergey Chernov, Information Retrieval Basics

Russian State Library5/24/2013 Sergey Chernov, Information Retrieval Basics

The MAT5/24/2013 Sergey Chernov, Information Retrieval Basics

Tate5/24/2013 Sergey Chernov, Information Retrieval Basics

The Hermitage5/24/2013 Sergey Chernov, Information Retrieval Basics

Outline Open Data Online and Social Media Data Sources Multimedia Archives Hands-on AssignmentSergey Chernov, Information Retrieval Basics

Types of OGD Apps 13 categories of appsidentified: health & safety, entertainment, sports, transportation, city services, real estate,5/24/2013 Sergey Chernov, Information Retrieval Basicshttp://www.slideshare.net/timoreilly/open-up-15195896 environment, education, public safety & lawenforcement, food & dining, development, business & finance, government & civics.

Russian Gov Data Applications Government tender system 3akupki.ru., bicotender.ru, initpro.ru, ist-budget.ru, is-zakupki.ru,goszakaz.ru, goszakupki-cpr.ru, marketing.interfax.ru,multitender.ru, my-tender.ru, regionzakaz.ru, seldon.ur.ru, tender-spb.ru Legal cases analysis rospravosudie.com, www.pravo.ru, www.intergrum.ru, spark-interfax.ru Income declarations declarator.org, publicprofit.ru Budget budget4me.ru, rosspending.ru Public transport rasp.yandex.ru, maps.google.ru, doroga.tv, Rusavtobus.ru5/24/2013 Sergey Chernov, Information Retrieval Basics

International OGD Apps http://www.fixmystreet.com/ http://seeclickfix.com/ http://codeforamerica.org/ http://open311.org/ www.recovery.gov http://data.seattle.gov/ (Notifire, MomMaps) https://nycopendata.socrata.com/ (WiFi) http://2011.nycbigapps.com/submissions http://daten.berlin.de/ (Wheelmap, Aircraft noise)5/24/2013 Sergey Chernov, Information Retrieval Basics

Hands-on assignment Working in teams of 2-3 people Search for motivating open data projects on the Web Explore data.mos.ru, openpolice.ru, opengovdata.ru orany other resource of your choice Propose a project based on some dataset. How the datalooks like? What is missing? Which skills and tools youmight need? Present your project5/24/2013 Sergey Chernov, Information Retrieval Basics