Date post: | 13-Dec-2014 |
Category: |
Documents |
Upload: | global-pulse |
View: | 3,846 times |
Download: | 3 times |
“BEYOND TARGETED ADS BIG DATA FOR A BETTER WORLD”
Robert Kirkpatrick Director, UN Global Pulse
O’Reilly Strata Conference | New York | October 2012
www.unglobalpulse.org @unglobalpulse
25 October 2012 | www.unglobalpulse.org
Photo Credit: John Oyuke
Kenyan Farm Workers
25 October 2012 | www.unglobalpulse.org
Hyperconnectivity
Airplane Flights
Internet Traffic
Telephone Calls
Social Networking
25 October 2012 | www.unglobalpulse.org
20th-Century tools…
25 October 2012 | www.unglobalpulse.org
PRIVATE SECTOR Monitor operations…in real time Track market trends…in real time
Get customer feedback…in real time
GLOBAL DEVELOPMENT Unemployment? Food Security? Public Health?
Education? Migration?
Disaster Relief?
1. Better early warning: Earlier detection of anomalies, trends and events allows earlier response.
2. Real-time awareness: A more accurate and up-to-
date picture of population needs supports more effective planning and implementation
3. Real-time feedback: Understanding sooner where
needs are changing -- or are not being met -- allows for rapid, adaptive course correction
BIG DATA IN REAL TIME: 3 OPPORTUNITIES
25 October 2012 | www.unglobalpulse.org
BIG DATA IS A HUMAN RIGHTS ISSUE
• Never analyze personally identifiable information • Never analyze confidential data • Never seek to re-identify individuals
25 October 2012 | www.unglobalpulse.org
RESEARCH BACKGROUND: A GROWING BODY OF EVIDENCE
How Mobile Phone Carriers See the World
Call Detail Records (CDRs) • Caller ID (hashed phone #) • Caller Tower Location • Receiver ID (hashed phone #) • Receiver Tower Location • Call Start Time • Call Duration
Airtime Expense Records • Caller ID (hashed phone #) • Caller Tower Location • Amount of Purchase • Time of Purchase • Balance at Time of Purchase
25 October 2012 | www.unglobalpulse.org
Modeling Behaviors in Mobile Data
Consumption variables • Number of calls, call duration, SMS/MMS/voice • Size, frequency, total number of airtime purchases • Handset Type and Features
Social variables • Degree of the social network • Weight of the contacts, frequency of communication
Mobility variables • Diameter of mobility and social network • Radius of gyration • Mobility Patterns
Source: Telefonica Research, 2011 25 October 2012 | www.unglobalpulse.org
EXAMPLE: AIRTIME CREDIT PURCHASE DATA
25 October 2012 | www.unglobalpulse.org
Higher household income
Lower household income
Average # of purchases / month
Aver
age
size
of p
urch
ase
SIZE AND FREQUENCY PREDICT HOUSEHOLD INCOME
25 October 2012 | www.unglobalpulse.org
Lower socioeconomic level
Higher socioeconomic level
CALLING PATTERNS AND ECONOMIC OPPORTUNITY
25 October 2012 | www.unglobalpulse.org
Men: • Fewer calls • Shorter calls • Smaller social network • More work-related calls
Women: • More calls • Longer calls • Larger social network • More personal calls
MEN AND WOMEN USE THEIR PHONES DIFFERENTLY
25 October 2012 | www.unglobalpulse.org
Tracking population movement to predict cholera
Source: Linus Bengtsson et. al., PLoS Medicine, 2011
25 October 2012 | www.unglobalpulse.org
A mobility index to evaluate H1N1 response in Mexico City
Source: Telefonica Research, 2011 See: http://www.unglobalpulse.org/publicpolicyandcellphonedata
25 October 2012 | www.unglobalpulse.org
dominican republic 11.7%
mexico 9.1% united kingdom 7.5%
canada 7.0%
guatemala 5.5%
ecuador 5.3%
jamaica 3.3%
india 3.0%
germany 2.5%
philippines 2.4%
TWITTER PREDICTS SPREAD OF INFLUENZA
r2 = .958
“You Are What You Tweet: Analyzing Twitter for Public Health. M. J. Paul and M. Dredze, 2011.” http://www.cs.jhu.edu/%7Empaul/files/2011.icwsm.twitter_health.pdf
25 October 2012 | www.unglobalpulse.org
Rumi Chunara et. al., American Journal of Tropical Medicine and Hygiene, 2012 86:39-45
25 October 2012 | www.unglobalpulse.org
GOOGLE SEARCHES FOR SYMPTOMS PREDICT DENGUE
25 October 2012 | www.unglobalpulse.org
2010 VS. 2011: INDONESIAN TWEETS ABOUT HIV
See: http://www.unglobalpulse.org/WorldAIDSDay-Part2 25 October 2012 | www.unglobalpulse.org
PROOF OF CONCEPT STUDIES
GLOBAL PULSE RESEARCH 2011
Online at: http://www.unglobalpulse.org/applyingbigdatatodevelopment
http://www.unglobalpulse.org/projects/can-social-media-mining-add-depth-unemployment-statistics
25 October 2012 | www.unglobalpulse.org
Online Discussions & Unemployment
Ireland
25 October 2012 | www.unglobalpulse.org
Online Discussions & Unemployment
United States
25 October 2012 | www.unglobalpulse.org
http://www.unglobalpulse.org/projects/twitter-and-perceptions-crisis-related-stress
Jakarta: nine million tweets per day
Map of Twi*er usage in Jakarta – by Eric Fischer 25 October 2012 | www.unglobalpulse.org
Tweets per day about food, during Ramadan in Indonesia
Start of Ramadan
End of Ramadan
25 October 2012 | www.unglobalpulse.org
Number of tweets per month commenting on the price of rice
Wordcloud1 (sunday)
Wordcloud2 (Sunday)
Food inflation
0.47
0.44
-0.85
1.49
2.81
2.21
-0.33
-1.94
-1.9
-0.28
1.27
1.84
1.07
-0.09
Data inflation
Tweets about the price of rice (per month)
Official Food Price Inflation (monthly from 25 cities)
Tweets predict food basket inflation (rice, chilies, fish, sugar, corn, cooking oil)
25 October 2012 | www.unglobalpulse.org
GLOBAL PULSE ABOUT
DIGITAL SERVICES AS HUMAN SENSOR NETWORKS: Observing fluctuations in well-being…in real-time
COPING STRATEGIES • Buy cheaper foods • Work longer hours • Reduce energy use • Draw down savings • Sell assets • Borrow from relatives
DIGITAL “SMOKE SIGNALS” • Depletion of airtime credit • Smaller mobile airtime
purchases • Failure to repay microloans via
mobile financial services • Changes in calling patterns • Inbound money transfers • Web searches for jobs, health • Sales of livestock via mobile
trading network • “Venting” on social media
25 October 2012 | www.unglobalpulse.org
Photo: Ministry of Foreign Affairs, Iceland
25 October 2012 | www.unglobalpulse.org
AGILE GLOBAL DEVELOPMENT?
25 October 2012 | www.unglobalpulse.org
• This data may be less accurate that official sources.
• But it’s faster. • And it’s cheaper to
collect. • How can we leverage
the speed to change the outcome?
Integrating real-time data into an institution
USGS Twitter Earthquake Detector (TED)
25 October 2012 | www.unglobalpulse.org
THE PROBLEM WITH TELESCOPES…
There’s a universe of data that we can’t see.
…AND MACROSCOPES
25 October 2012 | www.unglobalpulse.org
25 October 2012 | www.unglobalpulse.org
Data Philanthropy?
A global
real-time public/private
data commons?
EXAMPLE R&D PROJECT: Mobile Networks as Drought Sensors in the Sahel
Proposal • Obtain 2011-2012 mobile CDRs and airtime purchases. • Derive mobility, consumption, and social variables. • Correlate variables with precipitation levels, survey
data. • Identify signatures of drought impacts in 2011. • Identify signatures of aid impact in 2012. • Develop and evaluate prototype during next drought. • Release open source “appliance” through GSMA.
25 October 2012 | www.unglobalpulse.org
PULSE LABS
Joint Research | Rapid Prototyping | Capacity Building
25 October 2012 | www.unglobalpulse.org
Pulse Lab Network
Pulse Lab Jakarta…………October 2012 Pulse Lab Kampala……….January 2013 Other locations…………...???
25 October 2012 | www.unglobalpulse.org
PULSE LABS R&D INNOVATION STRATEGY
1. Partner with governments to establish Pulse Labs 2. Build world-class teams of data scientists,
engineers, and policy experts 3. Partner with private sector for real-time data and
cutting edge technology 4. Work with UN agencies and academia to conduct
research around challenges in 5. Build open source prototypes of tools to automatic
real-time monitoring 6. Support broad adoption of useful innovations 7. Share everything we learn and build
25 October 2012 | www.unglobalpulse.org
SO HOW DO I GET INVOLVED? ARE YOU.. • A company with powerful data you think
could make the world a better place? • A technology provider with screaming fast
computing or killer analytics? • A whiz data scientist interested in hard
problems, positive impact, and global scale? • A big data privacy expert who understands
that we cannot help unless we also protect? 25 October 2012 | www.unglobalpulse.org
25 October 2012 | www.unglobalpulse.org
Research Tool 1 Crimson Hexagon: ForSight
25 October 2012 | www.unglobalpulse.org
Food Prices: What a real crisis looks like
14 -‐ 21 Aug Ramadhan / Idul Fitri
23 July -‐ 2 Aug ‘tempeh’ and ‘tofu’ hot debate
25 October 2012 | www.unglobalpulse.org
Comparing Crises
23 July -‐ 2 Aug ‘tempeh’ and ‘tofu’ hot debate
during soybean shortage
18 Mar -‐ 7 Apr Fuel subsidy cut plan and
protests against it
Tweets about food
Research Tool 2 SAS Social Media Analytics and SAS Text Miner
25 October 2012 | www.unglobalpulse.org
Analytic Workflow
Internet Conversation
Global Pulse Relevance
Filter
Sentiment, Mood & Influence
Topic & Geography Categories
Interactive Dashboard
1) Over 200,000 new Indonesian language documents per day
3) Capture senKment and mood for Bahasa
2) Extract conversaKons about rice, cooking oil, fuel, employment, etc.
4) Detect locaKon, price, availability, specific govt. programs, etc.
5) Explore results and correlate with official staKsKcs to official BPS staKsKcs : Consumer Price Index (CPI) for 12 common foods
anxious, confident, confused, hosKle, sad, happy (-”-) ;-) ((+_+)) :@ :( :)
What’s the deal with Indonesians and eggs?
For every 5000 more tweets about eggs… …we see a 2-3% decrease in food CPI?
The Signals Are Getting Stronger
0
5000
10000
15000
20000
25000
30000
35000
40000
minyak (oil) ketahanan pangan (food security) budidaya (culKvaKon) telur (eggs)
è Big increase in volume of relevant conversations over 18 months
Indonesians are increasingly using social media to discuss basic needs
25 October 2012 | www.unglobalpulse.org
So Are the Temporal Correlations è Listening to social conversations provides insight on official data
-‐2.5
-‐2
-‐1.5
-‐1
-‐0.5
0
0.5
1
1.5
2
2.5
Social Media Food Index
BAPPENAS Food Price Index
25 October 2012 | www.unglobalpulse.org
Next up for Pulse Lab Jakarta research: 1 year of anonymized Indonesian CDRs
• 4 largest carriers • 170 million subscribers • 200 billion call records • 80 terabytes of data
25 October 2012 | www.unglobalpulse.org
25 October 2012 | www.unglobalpulse.org
Image credit: Aaron Koblin 24 hours of AT&T phone calls and Internet
traffic flowing through New York City
ROBERT KIRKPATRICK Director UN Global Pulse www.unglobalpulse.org [email protected] +1 (650) 796-5709