Juliana Freire PPT

transcript

Exploring Big and not so Big Data: Opportunities and Challenges

Juliana Freire juliana.freire@nyu.edu

Visualization and Data Analysis (ViDA) Center http://bigdata.poly.edu

NYU Poly

2 ViDA Center Juliana Freire

Big Data: What is the Big deal?

http://www.google.com/trends/explore#q=%22big%20data%22!

  Many success stories –  Google: many billions of pages indexed, products,

structured data –  Facebook: 1.1 billion users using the site each month –  Twitter: 517 million accounts, 250 million tweets/day

  This is changing society!

  Smart Cities: 50% of the world population lives in cities –  Census, crime, emergency visits, cabs, public transportation,

real estate, noise, energy, … –  Make cities more efficient and sustainable, and improve the

lives of their citizens http://www.nyu.edu/about/university-initiatives/center-for-urban-science-progress.html

  Enable scientific discoveries: science is now data rich –  Petabytes of data generated each day, e.g., Australian radio

telescopes, Large Hadron Collider –  Social data, e.g., Facebook, Twitter (2,380,000 and 2,880,000

results in Google Scholar!)   Data is currency

  Smart Cities –  Census, crime, emergency visits, cabs, public transportation,

real estate, noise, energy, … –  Make cities more efficient and sustainable, and improve the

lives of their citizens   Enable scientific discoveries: science is now data rich

–  Petabytes of data generated each day, e.g., Australian radio telescopes, Large Hadron Collider

–  Social data, e.g., Facebook, Twitter

  Data is currency

  Big data is not new: financial transactions, call detail records, astronomy, …

  What is new is that there are many more data enthusiasts

  More data are widely available, e.g., Web, data.gov, scientific data

  Computing is cheap and easy to access –  Server with 64 cores, 512GB RAM ~$11k –  Cluster with 1000 cores ~$150k –  Pay as you go: Amazon EC2

Astronomy

Geosciences

Chemistry Microbiology

2010 Social Sciences

Physics

Medicine

Plot from Howe and Halperin, DEB 2012

  Big data is not new: financial transactions, call detail records, astronomy, …

  What is new is that there are many more data enthusiasts

  More data are widely available, e.g., Web, data.gov, scientific data, social and urban data

  Computing is cheap and easy to access –  Server with 64 cores, 512GB RAM ~$11k –  Cluster with 1000 cores ~$150k –  Pay as you go: Amazon EC2

Big Data: What is hard?

  Scalability is not the problem…   Usability is the Big issue

data knowledge

statistics

algorithms

machine learningmath

user interfaces

data visual encodings

interaction modes

technology

data management

provenance

Exploring data is hard

data knowledge

statistics

algorithms

user interfaces

interaction modes

technology

data management

provenance

Exploring data is hard, regardless of whether the data

is big or small

data knowledge

statistics

algorithms

user interfaces

interaction modes

technology

data management

provenance

Case Study: Studying Cab Trips in NYC

Prepare data for analysis   Raw data for 2011 63 GB

–  24 csv files, 2 csv files for each month - one for trip data, and snother for fare data

–  ~170M trips

  Cleaning –  ~60,000 fare records do not have trip records –  ~200 duplicates per month

Storage Solutions: Temporal Queries

  SQLite – 20 GB of storage

(index on pickup_time)

– Ordered queries: 9.39s

– Reverse ordered queries: 9.41s

– Shuffled queries: 9.37s

  Custom storage – 12 GB of storage (in-

memory binary search instead of index)

– Ordered queries: 0.6s – Reverse ordered

queries: 1.4s – Shuffled queries: 1.2s

Storage Solutions: Spatial-Temporal

  All trips for a week in a given region   All trips in a week for a given taxi   All trips in a week for a given taxi in a

given region

Needs a complex indexing scheme that combines spatial, temporal, and taxi id searches

Storage Solutions: Spatial-Temporal

  SQLite – 20+10 GB of storage

(index on time and id, r-tree for coordinates)

– Creating indexes: 52hrs

– Range queries: 2.1s – Combined queries:

15.3s – Cross-table queries:

  Custom storage (ours) – 12+4 GB of storage

(using (4d) kd-tree on time, id and coordinates)

– Building kd-tree: 8 mins

– Range queries: 0.2s – Combined queries:

0.2s – Cross-table queries:

Summary Statistics

  13,237 Medallion Cabs   42,000 Taxi Drivers   Average Number of Rides: 485k/day   Average Number of Passengers: 660k/day

Analysis/Modeling

Rides in 2011

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Aug 28 Irene

Apr 2 Apr 3

Dec 25

Rides per Hour June 2011

Between 5k and 35k rides/hour

Rides at Midnight

Night Life!

Weekly Patterns

Analysis/Modeling

TLCVis

Drop-off

Pickup

Most of the drop-off’s occur on the avenues while most of the pick-up’s occur on the streets

Drop-offs vs. Pickups

Studying Anomalies

8:00AM-8:30AM 6:00AM-6:30AM 4:00AM-4:30AM

Sunday, May 1st 2011

Studying Anomalies

8:00AM-8:30AM 6:00AM-6:30AM 4:00AM-4:30AM

Sunday, May 1st 2011

Studying Anomalies

8:00AM-8:30AM 9:30AM-10:00AM Sunday, May 1st 2011

Studying Anomalies

8:00AM-8:30AM 9:30AM-10:00AM Sunday, May 1st 2011

Five Borough Bike Tour

Interpretation

Studying Anomalies

Sunday May 1st 2011

07:00AM-08:00AM

Studying Anomalies

Sunday May 1st 2011

08:00AM-10:00AM

Studying Anomalies

Sunday May 1st 2011

10:00AM-11:00AM

Studying Patterns

May 1st – May 7th 2011

3.6 Million Trips

Compare movement in the

airports against the large train stations

Studying Patterns

3.6 Million Trips

Train Stations Airports

Studying Patterns

3.6 Million Trips

Train Stations Airports

Data exploration reveals bad data…

Uses of Clean Data: FindMeACab App

Take Away

  Data exploration is challenging for both small and big data

  It is hard to prepare data for exploration   For many tasks, existing tools are either too

cumbersome, not scalable, etc.   Need better, usable tools

–  Tools for data enthusiasts who are not computer scientists!   Visualization is essential for exploring large volumes

of data --- “A picture is worth a thousand words’’   Pictures help us think [Tamara Munzner]

–  Substitute perception for cognition –  Free up limited cognitive/memory resources for higher-

level problems

Masters in Big Data

  New degree at NYU Poly – Spring 2014   Courses:

–  Machine learning –  Massive data analysis –  Visualization –  Visual Analytics –  Database Systems –  Algorithms –  …

Thanks

Juliana Freire PPT

Documents