Data streaming algorithmsSandeep JoshiChief hacker

1

Problem Statement

In limited space, in one pass, over a sequence of items

Compute the following

min, max, average,

standard deviation

moving average

Cardinality (count of distinct items in a stream)

Heavy hitters (aka find most frequent items)

Order statistics (rank of an item in sorted sequence)

Histogram (frequency per item)

2

Space-time axis

3

Space

Time

N N^2 N^3 exp

N

N.logN

logN

N^k

DeterministicAnd

Randomizedalgorithms

Linear time

Our focus : Linear time (preferably one pass) & Randomized

exp

Approach

• Will present simplified algorithms to provide general idea.• Not going to cover all proposed solutions for a problem.• Sacrifice rigor to provide intuition.

4

Not going to cover

• Sampling techniques• Case where input is sequence of strings or multi-dimensional• Set membership problem (bloom filters, etc)• Outlier detection• Time series-related algorithms• How to extend algorithms to distributed setting

5

1. Cardinality

6

Bits emitted by a hash

In hash of all items, observe number of times you get bit ‘1’ followed by many zeros

7

Bit patterns

For num = [1, 1000] h = hash(num)

Number of hashes ending in Out of 1000

0 530

10 281

100 140

1000 53

10000 28

100000 9

1000000 12

10000000 5

100000000 2

1000000000 0

10000000000 0

100000000000 0

8

Bit ‘1’ followed by 9 or more zeroes not foundBecause 1000 ~ 2^10

Flajolet-Martin sketch algo

1. For each item2. Index = rightmost bit in hash(item)3. Bitmap[index] = 1(at this point, bitmap = “000...00000101011111”)4. Estimated N ~ 2 rightmost ‘0’ bit in bitmap

9

Further improvements : split stream into M substreams and use harmonic mean of their counters, use 64-bit hash instead of 32, add custom correction factors to hash at low and high range.

Why it works

• The number of distinct items can be roughly estimated by the position of the rightmost 0-bit.

• A randomized algorithm which takes sublinear space - number of bits is equal to log2(n)

• Algorithm also works over strings [ 1985 paper uses strings ]• Any set of bits can be used [ hyperloglog uses middle bits]

10

Comparison between 3 different versions

* my FM-sketch implementation is incomplete – actual algo is not that bad

11

X : actual cardinality

Y : estimatedcardinality

What is a sketch ?

• A sketch maintains one or more “random variables” which provide answers that are probabilistically accurate.

• In Hyperloglog, this random variable is the “position of the rightmost zero”. It roughly estimates the actual cardinality of the set.

• A sketch uses universal hash function to distribute data uniformly.

• To reduce variance, it may use many pairwise-independent hashes and take their average.

12

* all random variables do not have normal distribution. Above Pic is to help in visualizing

2. Heavy Hitters

13

Heavy Hitters problem

• Find the items in a sequence which occur most frequently• We will see two algorithms 1. Karp, Shenker and Papadimitrou2. Count-Min sketch by Cormode and Muthukrishnan. Versatile algo

which has many applications

14

Heavy Hitters – Karp, et al

1. Keep a frequency Map<item, count>2. For each v in sequence3. increment Map[v].count4. If map.size() > threshold5. for each element in Map6. decrement Map[element].count7. if count is zero, delete Map[element]

Algo has second pass to adjust counts. Paper discusses additional optimizations. Implemented in Apache Spark. See DataFrameStatFunctions.freqItems().

Maintain a truncated histogram

15

Count-Min sketch

http://stackoverflow.com/questions/6811351/explaining-the-count-sketch-algorithm

To find frequency of an item, get minimum value in all ‘d’ slots that item that item got hashed to.Since many items could have incremented the same slot (one-sided error), using ‘min’ instead of ‘average’ is better.

Count-Min Sketch applications

• For heavy hitters, need additional heap data structure to maintain those items which hashed to high value slots.

• Point query• Range query using dyadic ranges• Joins• Temporal extension (Hokusai) to store historical sketches at lower

resolution.

17

3.Order statistics

18

Order statistics terminology

Given sorted sequence [1, 1, 1, 2, 3]

1. 0-quantile = minimum 2. 0.25 quantile = 1st quartile = 25 percentile3. 0.50 quantile = 2nd quartile = 50 percentile = median4. 0.75 quantile = 3rd quartile = 75 percentile5. 1-quantile = maximum

19

Order statistics offline algorithm

• There exists an offline and exact algorithm to find the kth item in a set• QuickSelect (Blum, et al) which is effectively a truncated quicksort• Can run in linear time algorithm (depending on pivot)

20

Pic : http://codingrecipies.blogspot.in/

Frugal streaming

1. Median_est = 0

2. For v in stream

3. if (v > median_est)

4. Increment median_est

5. else if (v < median_est)

6. Decrement median_est

21

Memory = log(N) bits where N = cardinalityCaveat: Reported median may not be in the streamPerforms poorly on sorted dataWorks best if stream items are independent and randomMedian drift s in the direction of the true median.Probability of drifting after reaching true median is low.Paper discusses extension to compute other quantiles

4 2 1 5 52 43

4 4 2 4 33 43

2 1 2 32 43

Stream

True median

estimated 1

T-Digest - Dunning et al

22

Each centroid attracts points nearest to it. Keeps “average” and “count” of these points.Maintain a balanced binary tree of centroid nodes

T-Digest for quantile

• Use sorted structure to find quantiles.• Centroids at both ends are deliberately kept small to increase accuracy of

outliers. • Can merge two T-digests.• Performs poorly on ascending/descending stream.

23

4. Histogram

24

Histogram

Two major problems1. How to decide bucket ranges apriori when data is being inserted in

unsorted order.2. What count should be returned in case of a partial bucket.

25

Sum & difference game2 4 10 18 6044 6640

3 14 42 63 -1 -4 -2 -3

8.5 52.5 -5.5 -10.5

30.5 -22

30.5 -22 -5.5 -10.5 -1 -4 -2 -3

original

transform

Sum & difference

Sum & difference game2 4 10 18 6044 6640

3 14 42 63 -1 -4 -2 -3

8.5 52.5 -5.5 -10.5

30.5 -22

30.5 -22 -5.5 -10.5 -1 -4 -2 -3

original

transform

Sum & difference

3 3 14 14 6342 6342

30.5 -22 -5.5 -10.5 0 0 0 0 Throw away small coefficients to get approximation

Histogram is approximated2 4 10 18 6044 6640

3 3 14 14 6342 6342

Wavelet based histograms

• Matias, et al. used this idea to store a compressed version of original frequency counts.

• Range query : to find counts within a range (e.g. 1 < x < 4), you need only “green-color” coefficients instead of all.•Original algorithm was applied on cumulative (CDF) instead of PDF; used linear wavelet instead of Haar, and had sophisticated thresholding to eliminate some wavelet coefficients.

29

2 4 10 18 6044 6640

3 14 42 63 -1 -4 -2 -3

8.5 52.5 -5.5 -10.5

30.5 -22

30.5 -22 -5.5 -10.5 -1 -4 -2 -3

Time vs frequency domain

Time domain view Frequency domain viewPic; https://e2e.ti.com/

Sometimes easier to solve problems in frequency domain

References

• Blog : https://research.neustar.biz/tag/streaming-algorithms/• Code : http://github.com/clearspring/stream-lib• Code : http://github.com/twitter/algebird• Book : Ullman et al, Mining Massive Data sets• Gist : http://gist.github.com/debasishg/8172796

31

Backup

K-min values for cardinality Munro-Paterson : median cannot be calculated exactly without O(n) memory. Similar result for cardinality and heavy-hitters.Wavelet : transform takes O(N), thresholding takes O(N.logN.logm), query takes O(m) where m = truncated coeff, N = original data.

Histogram from various perspectives

• Statistics : known as “density estimation”. Its non-parametric because we are not told how points are distributed ahead of time. Two approaches

1) parzen windows2) nearest neighbour (k-means).

• Computer science : k-segmentation problem; solved with Bellman’s dynamic programming algorithm.

• Signal processing : translate time domain problem into frequency domain.

33