Designing for Massive Scalability at BackType #bigdatacamp

transcript

Designing for Massive Scalability at BackType

Michael Montano / @michaelmontano

Desired properties of a back-end

Wednesday, November 17, 2010

• Robust and fault-tolerant to both machine and human error.

• Low latency reads and updates.

• Scalable to increases in data or traffic.

• Extensible to support new features or related services.

• Generalizes to diverse types of data and requests.

• Allows ad hoc queries.

• Minimal maintenance.

• Debuggable: can trace how any value in the system came to be.

Layered Architecture

Speed layer

Batch layer

Serving layer

Layered Architecture

Speed layer

Batch layer

Serving layer

Work in tandem to satisfy our desired properties

Batch Layer

view = fn(complete dataset)

Batch Layer Views

• Arbitrary

• High latency

• No random access

Serving Layer

• Provide random access to batch-computed views

• Update in batch, no random writes

• High latency updates

ElephantDB

• Our implementation of serving layer

• Pre-shard key/value data via MapReduce

• ElephantDB ring pulls shards from HDFS on startup

• Read-only access to data

ElephantDB

Shards on HDFS

Batch Layer

ElephantDB Flow

Batch and Serving Layers

Complete dataset (HDFS)

Tweet count view

Influencer scores view

Site affinity view

Batch Layer Serving Layer

ElephantDBShards

ElephantDBRing

Batch and Serving Layers

Robust and fault-tolerant to both machine and human error. Low latency reads and updates.

Scalable to increases in data or traffic.

Extensible to support new features or related services.Generalizes to diverse types of data and requests.

Allows ad hoc queries.

Minimal maintenance.

Debuggable: can trace how any value in the system came to be.

Speed Layer

• Compensate for high latency of updates to serving layer

Speed Layer

Key point: Only needs to compensate for data not yet absorbed in serving layer

Speed Layer

Key point: Only needs to compensate for data not yet absorbed in serving layer

Hours of data instead of years of data

Application-level Queries

Serving Layer

Speed Layer

• Speed layer is transient

• Serving layer eventually corrects speed layer

• Can make tradeoffs aggressively for performance

• Can even tradeoff accuracy

ExampleExample: Unique visitors to a domain

• Batch/Serving layers

• Compute exact count

• Speed layer

• Keep set of visitors in a bloom filter

• Incrementally update count and bloom filter

Designing for Massive Scalability at BackType #bigdatacamp

Documents