Particle CollisionIn Code Space

API meets Big Data

Chris Boos (@boosc)boos@arago.de

How to Web, Bucarest 8.11.2012

Mittwoch, 14. November 12

Big Data?

Mittwoch, 14. November 12

Data, lots of itMittwoch, 14. November 12

Even in simple datasets, common statistics fails - (avg, min, max, distribution)

Mittwoch, 14. November 12

Wisdome2010s Systems Thinking

Intelligence2000s Knowledge Ecology

Knowledge1990s Knowledge Management

Why you need big data

Yield

Data1960s1950s Data Processing

You Are Here !

Information1980s1970s Information Management

Mittwoch, 14. November 12

Because, (now) We Can!

Mittwoch, 14. November 12

79 times more CPU power than used in Apollo missions on one iPhone 4

Mittwoch, 14. November 12

All the resources you like at your fingertips

Mittwoch, 14. November 12

Buzzword Bingo

Big DataData Engineer

H-Space

HadoopCassandra HBasePIGredis.io Eucalyptus

Machine Learning Support Vector Machines

Gaussian ProcessesSwarm Intelligence

Genetic Algorithms

Agents/Bots

R+Natural Language Processing

ClusteringCore Dataset

NoStats

Mittwoch, 14. November 12

Data Exploration Is a Big Adventure

Mittwoch, 14. November 12

Big Data is all about improving prognostic power - guess what it is NOT!

What people think

Mittwoch, 14. November 12

Finding clusters, evaluating outliers and interpreting white noise

Warming up

Mittwoch, 14. November 12

You are not looking for patterns, you are looking for anomalies

Mittwoch, 14. November 12

Two tips for looking at data:

1. Plot it

2. Remove all labels

Mittwoch, 14. November 12

3 Ways to solve a coding or

data problemElegant

Standard

Brute Force

Mittwoch, 14. November 12

The Sledge Hammer of Big Data - Map Reduce

Mittwoch, 14. November 12

Old Style (Imperative) Programming

• Step by step explanation what to do

• Explaining WHAT to do rather than RESULTS you want

• Always necessary for basic algorithms

1

2

3

Mittwoch, 14. November 12

One New Style (Functional) Programming I

• Combine results to become a program

• Allows dynamic distribution

• Map-Reduce is only one way of doing it!

1

2

3

Mittwoch, 14. November 12

Functional Programming II

F ( G ( H ( A,B) , C), D)

getMusicLikes(getFriends(facebookID)

Instead of

for i in getFriends(facebookID) getMusicLikes(i)

Mittwoch, 14. November 12

BASE(Basically Available, Soft State, Eventual consistency)

not

ACID(Atomicity, Consistency, Isolation, Durability)

Mittwoch, 14. November 12

That is all batch processing!?

Mittwoch, 14. November 12

The big in memory trend is delaying the problem

Mittwoch, 14. November 12

Google is going well beyond Map Reduce and batch processing

Mittwoch, 14. November 12

And at facebook they are going far beyond distributed data storage and processing

Mittwoch, 14. November 12

Most people using Big Data end up building their own platform

Mittwoch, 14. November 12

APIs, for external use only?

NO WAY!

Mittwoch, 14. November 12

Monolithic architectures are so out!!!

Mittwoch, 14. November 12

Do not use Design Patterns as an excuse to get lost in OO space!!

Mittwoch, 14. November 12

Standardising dev tools can mean 2 things:

You are a little dictator and could not find a country

Your team is not as good as you say and needs guidance

Mittwoch, 14. November 12

API 1.0

Mittwoch, 14. November 12

Function calls or servicesMittwoch, 14. November 12

Stop religion on transfer format (XML, JSON), who cares?

Mittwoch, 14. November 12

Stop religion on service design (SOAP, REST), be practical!

Mittwoch, 14. November 12

API 2.0

Mittwoch, 14. November 12

Adding semantics means adding context

Mittwoch, 14. November 12

OWL or the theoretic vision of everyone speaking the same language

and meaning the same thing

Mittwoch, 14. November 12

Facebook‘s pragmatic approach: the open graph, sort of RDF

Mittwoch, 14. November 12

API Management and Monetization

Mittwoch, 14. November 12

oAuth vs. application tokens - or both?

Mittwoch, 14. November 12

Monetization models

API calls

Data transfer

Data access

Mittwoch, 14. November 12

Not necessarily DIY, why not use an external API manager?

Mittwoch, 14. November 12

CollisionBIG Data and APIs

Mittwoch, 14. November 12

Your platform cannot handle infinite exploration

Mittwoch, 14. November 12

Your users will not understand the mechanics behind your data processing

Mittwoch, 14. November 12

Your data has to be put into context

Mittwoch, 14. November 12

Then you can onboard new team members quickly and get external developers to use your stuff

Mittwoch, 14. November 12

Check out my tool list:http://www.hcboos.net/100-links/

Mittwoch, 14. November 12

Credits• Images on pages 8,11,15,21,22,25,29,31,32,33,34,39,40,41,43,44,45,46

are used with kind permission of CERN, are (c) CERN and are used under the following terms of use: http://copyright.web.cern.ch/

• „Big Data Just Beginning to Explode“ by CSC http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode

• „Social media network connections among twitter users“ by Marc Smith http://www.flickr.com/photos/marc_smith/

• Asteroid Datasets by Bruce Gary http://brucegary.net/POVENMIRE/x.htm

• iPhone Images (c) Apple Computer inc.

• Google Percolator chart by Marc Bojoly http://blog.octo.com/en/my-reading-of-percolator-architecture-a-google-search-engine-component/

• Prism image by Deviant Art http://sudden2.deviantart.com/art/Floyd-s-Prism-51863247

• Monolith image found on http://www.themonolith.com/

• Design Patterns: Elements of Reusable Object Oriented Software, Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, 1994, Addison Wesley, ISBN 0-201-63361-2

• OWL Logo (c) World Wide Web Consortium http://www.w3c.org

Mittwoch, 14. November 12

Thank You for Your Time

Chris Boos (@boosc)boos@arago.de

How to Web, Bucarest 8.11.2012

Mittwoch, 14. November 12

2 Examples

Mittwoch, 14. November 12

The AMP3 Platform at Senzari.comAdaptable Music Parallel Processing Platform

Mittwoch, 14. November 12

MARS-o-Matic at arago.deBig Data based IT modelling and pricing app

More Info at www.mars-o-matic.com

Mittwoch, 14. November 12