Real-time “OLAP” for Big Data (+ use cases) - bigdata.ro 2013

transcript

Cosmin Lehene | Adobe

#bigdataro - 30 January 2013

Real-time “OLAP” for Big Data (+ use cases)

What we needed … and built

OLAP Semantics Low Latency Ingestion High Throughput Real-time Query API

“Physical” Building Blocks

Logical Building Blocks

Dimensions, Metrics Aggregations Roll-up, drill-down, slicing and dicing, sorting

OLAP 101 – Queries example

Date Country

City OS Browser Sale

2012-05-21

USA NY Windows FF 0.0

2012-05-21

USA NY Windows FF 10.0

2012-05-22

USA SF OSX Chrome 25.0

2012-05-22

Canada Ontario Linux Chrome 0.0

2012-05-23

USA Chicago OSX Safari 15.0

5 visits,3 days

2 countriesUSA: 4Canada: 1

4 cities:NY: 2SF: 1

3 OS-esWin: 2OSX: 2

3 browsersFF: 2Chrome:2

50.03 sales

OLAP 101 – Queries example

Rolling up to country level:

SELECT COUNT(visits), SUM(sales)

GROUP BY country

“Slice” by browser

SELECT COUNT(visits), SUM(sales)

GROUP BY country

HAVING browser = “FF”

Top browsers by sales

SELECT SUM(sales), COUNT(visits)

GROUP BY browser

ORDER BY sales

Country visits

USA 4 $50

Canada 1 0

Country visits

USA 2 $10

Canada 0 0

Browser sales visits

Chrome $25 2

Safari $15 1

FF $10 2

Aggregate at runtime Most flexible

Fast – scatter gather

Space efficient

But I/O, CPU intensive

slow for larger data

low throughput

Pre-aggregate Fast

Efficient – O(1)

High throughput

But More effort to process

(latency)

Combinatorial explosion (space)

No flexibility

OLAP – Runtime Aggregation vs. Pre-aggregation

SaasBase Map

SaasBase Domain Model Mapping

SaasBase - Domain Model Mapping

SaasBase - Ingestion, Processing, Indexing, Querying

Ingestion

Ingestion(ETL) throughput vs. latency

Historical data (large batches) Optimize for throughput

Increments (latest data, smaller) Optimize for latency

Processing

Processing involves reading the Input (files, tables, events), pre-aggregating it (reducing cardinality) and generating cubes that can be queried in real-time

“Super Processor” code running in Storm, Map-Reduce, HBase

Processing for OLAP semantics

GROUP BY (process, query)

COUNT, SUM, AVG, etc. (process, query)

SORT (process, query)

HAVING (mostly query, can define pre-process constraints)

SaasBase vs. SQL Views Comparison

Query Engine

Always reads indexed, compact data

Query parsing

Scan strategy

Single vs. multiple scans

Start/stop rows (prefixes, index positions, etc.)

Index selection (volatile indexes with incremental processing)

Deserialization

Post-aggregation, sorting, fuzzy-sorting etc.

Paging

Custom dimension/metric class loading

Adobe Business Catalyst

Online business presence: e-commerce, marketing, web analytics etc.

Use case: Web Analytics (visitors, channels, content, e-commerce, campaigns, etc.)

BC - Workflow

Adobe Business Catalyst - Stats

3 active datacenters

Raw data ~6TB (from ~1TB 18 months ago)

Visits table: ~1TB each(compressed)

OLAP cubes (stats): 49GB – 64GB (compressed)

~30 minutes latency (from actual pageview/sale to chart in UI)

10s – 100s of milliseconds latency for queries

~3000/s max concurrent OLAP queries (actual traffic is much lower)

Adobe Pass for TV Everywhere

Authentication & Authorization

Single sign-on to Programmer content (e.g. Turner, NBC, Hulu, MTV, etc) with Cable operator credentials (e.g. Comcast, Dish, etc.)

Adobe Pass – Use Case

Analytics use case: Operational metrics (users, devices, latencies, etc.)

Real-time ingestion in HBase

High Frequency Map Reduce jobs (every 2 minutes)

Adobe Pass - Stats (London Olympics 2012)

67M streams ~ 5.3M hours

1.5M concurrent streams

> 7M unique users

1 Technical & Engineering Emmy Award ;)

Adobe Primetime – Real-time Video Analytics

Unified video platform (acquisition, transcoding, broadcast, ads, analytics)

Adobe Primetime – Use Case

Use Cases: Audience metrics – minutes latency ok

Ads metrics – seconds to minutes ok

Streaming QoS metrics – seconds must

Requirements: Massive throughput (millions of streams, multiple

heartbeats every 10 seconds)

Low latency (end-to-end)

Conclusions

OLAP semantics on a simple data model

Data as first class citizen

Domain Specific “Language” for Dimensions, Metrics, Aggregations

Framework for vertical analytics systems

Tunable performance, resource allocation

Thank you!Cosmin Lehene @clehene

http://hstack.org

http://www.slideshare.net/clehene/low-latency-olap-with-hbase-hbasecon-2012

Real-time “OLAP” for Big Data (+ use cases) - bigdata.ro 2013

Technology