© Copyright StreamBase®. Proprietary & Confidential.www.streambase.com 1

One Size Fits All: An Idea Whose Time has Come and Gone

Michael Stonebraker

www.streambase.com 2

Alternate TitleAlternate Title

The elephants are selling 30 year old “bloatware”

That is not good at anything

And you should send them to the “home for old software”

www.streambase.com 3

Three Financial Services MarketsThree Financial Services Markets

Stream processing (electronic trading)

Tick stores (data warehouses)

OLTP (transaction processing)

www.streambase.com 4

Stream ProcessingStream Processing(Electronic Trading)(Electronic Trading)

A feed comes out of the wall

Compute a “secret sauce” looking for events of interest

Trade based on the result

But only if you are more nimble than the next guy….

www.streambase.com 5

Traditional RDBMS ModelOutbound Processing

Store the data before processing!

LatencyWhat if the data is not important?

Too many processes! Optimized for business data

processing Where you don’t trust the app.

Queries

Memory

Disk

Updates

Processing

Too slow to be interesting!

www.streambase.com 6

Stream Processing Engine with StreamSQL

Database paradigm (SQL) a good one

But need a different architecture

Straight through processing

No task switches

Lightweight scheduling

Inbound Processing

Memory

Disk

StreamBase Application

Event Data

Queries

Alerts Actions

Alerts Actions

Streambase Application

www.streambase.com 7

• Example: Every minute for every stock I am trading: Calculate VWAP (vol. weighted avg. price) for my trades & all trades Alert whenever my personal trading execution is inferior to market

5 Streambase operators, 30 min to build Streams of “tuples” (time-series data) flow through query

Queries run continuously

StreamSQL Application Example

”

Market_Feeds

My_Buys

Alerts

www.streambase.com 8

StreamSQL Will Dominate Rule Engines

Essentially all applications entail a mix of stored and real-time dataStreamSQL covers both kinds of data in a single paradigmA rule engine must switch paradigms

StreamSQL amenable to compilationKnow what is the next event to processIn contrast, hard to figure this out in a rule engine

www.streambase.com 9

Performance Benchmark

Financial Services Application:

Construct a virtual feed of “first arrivers” on a low end Linux machine

Relational DB: 11,000 messages/secStreambase: 300,000 messages/secAnother StreamSQL vendor: 20,000 messages/sec

Result: Streambase was a factor of 27 faster

www.streambase.com 10

Tick Stores Tick Stores (and Other Warehouse Applications)(and Other Warehouse Applications)

Store all market data for the last 10 years

To back test “secret sauce” models

To answer ad-hoc queries – “how many times has X happened”

Typical size – 100 Tbytes

Append only

www.streambase.com 11

Terminology -- “Row Store”

Record 2

Record 4

Record 1

Record 3

E.g. DB2, Oracle, Sybase, SQLServer, …

www.streambase.com 12

Rotate Your Thinking 90 Degrees Rotate Your Thinking 90 Degrees

Column stores read only the columns required

Not all of them

Compression works better

By a factor of 2-3 against the elephants

No record headers

Which are big ticket items

No padding to byte or word boundaries

www.streambase.com 13

Benchmark SummaryBenchmark Summary

Vertica has been baked off about 30 times

Typically against the incumbent

Has yet to win by less than a factor of 30 against a

row store

Beats most other column stores by around 10X

KX is the only system to come within an order of

magnitude

www.streambase.com 14

Maybe Elephants are Good Maybe Elephants are Good at OLTP……at OLTP……

OLTP is a main memory market

Not a disk-based one

Transactions are short and have no I/O or user stalls

Run to completion (single threaded)

Disaster Recovery (and HA) a requirement

Build it into the bottom of the system

www.streambase.com 15

TPC-C Performance TPC-C Performance on a Low-end Machineon a Low-end Machine

Elephant

850 TPS (1/2 the land speed record per processor)

H-Store (so far – a university prototype)

70,416 TPS (41X the land speed record per processor)

Factor of 82!!!!!

www.streambase.com 16

Implications for the ElephantsImplications for the Elephants

They are selling “one size fits all”

Which is 30 year old legacy technology that is good at nothing

www.streambase.com 17

Pictorially:

OLTPData Warehouse

Streaming data

DBMS apps

www.streambase.com 18

The DBMS Landscape – Performance Needs

OLTPData Warehouse

Streaming data

low

high

high

high

www.streambase.com 19

One Size Does Not Fit All -- Pictorially

Open source

Vertica H-Store

successors

StreambaseElephants get only “the crevices”

© Copyright StreamBase®. Proprietary & Confidential.www.streambase.com 20

Thank You

Member

Corporate Headquarters181 Spring StreetLexington, Massachusetts 02421+1 866 STRMBAS+1 866 787 6227+1 781 761 0800

New York City Office220 West 42nd Street, 20th FloorNew York, New York 10036+1 866 STRMBAS+1 866 787 6227

Reston, Virginia Office11921 Freedom Drive, Suite 550 Reston, VA 20190+1 703 608 6958

London Office107-111 Fleet StreetLondon EC4A 2ABUnited Kingdom+44 (0)20 7936 9050