Foto Afrati — National Technical University of Athens Anish Das Sarma — ClearList Inc .

transcript

Foto Afrati — National Technical University of Athens

Anish Das Sarma — ClearList Inc.Anand Rajaraman — Cambrian Ventures

Pokey Rule — Stanford UniversitySemih Salihoglu — Stanford University

Jeff Ullman — Stanford University

Anchor Points Algorithms for Hamming and Edit Distance

Fuzzy Joins

Input: set of records ROutput: <reci, recj> pairs s.t. dist(reci, recj) ≤ d

…recm

Input Output<rec1, rec5><rec7, rec9>

…<rec3, reck>

Example Applications: entity resolution, clustering, collaborative filtering

Two Specific Distance Measures

1. Hamming Distance Input: bit strings R of length n

2. Edit Distance Input: strings R of length n over alphabet A

0000000001

…10011

<00000, 00001>

…<10011, 10010>

eabc…

<abcd, eabc>

…<dddd, dadd>

Fuzzy Joins In One-Round MapReduce

recm-1

values

rec1, rec5, rec7

rec2, rec7, recm

rec2, recm

Reduce

reducer1

reducer2

reducerp

Per-Reducer-Memory-Cost

Communication Cost

communication

|R|=2n

2 |R|=2n

Grouping

(naïve)

per-reducer memory

2n-d+1

Ball-Hashing

O(nd/2)

Splitting

Communication Cost vs Per-reducer Memory

Anchor Points

Outline

1. Anchor Points Algorithm

• Covering Code

2. Explicit Construction of Hamming Distance Covering

3. Explicit Construction of Edit Distance Covering Codes

Outline

• Covering Code

2. Explicit Construction of Hamming Distance Codes

3. Explicit Construction of Edit Distance Codes

Covering Code

Given set of strings R of length n, and radius k Definition: <n, k> covering code C

for each s∈R, there is a c∈C, s.t dist(c, s) ≤ k

kn length of stringsd distance of pairsk radius of code

Example Covering Code

01111 … 11101 11110

00111 … 10011 … 11100

00011 00101 … 10001 11000

00001 … 01000 10000

Example: Hamming Distance, n=5, k = 2

… …

……

… …

…n length of stringsd distance of pairsk radius of code

00000010000101101100

…1111011111

Map Reduce

Let C be an <n, k> covering code => (e.g. n=5, k=2)One reducer for each code wordMap s to code words at distance ≤ k + d/2 => (e.g. d=2 => 2 + 2/2 = 3)

Anchor Points Algorithm (1)

r00000

r11111

Anchor Points Algorithm (2)

≤d/2

w≤d/2

≤k + d/2≤k + d/2

Triangle Inequality

n length of stringsd distance of pairsk radius of code

Cost of Anchor Points Algorithm

B(n, r): size of the ball of radius rPer-reducer memory: B(n, k + d/2)Communication: |C|B(n, k + d/2)

Reducer for code word c

k + d/2s4

communication

|R|=2n

2 |R|=2n

Groupin

g (naïve)

per-reducer memory

2n-d+1

Ball-Hashing

O(nd/2)

Splitting

Anchor Pointsk=0

Communication Cost vs Per-reducer Memory

Outline

• Covering Code

2. Explicit Construction of Hamming Distance Codes

3. Explicit Construction of Edit Distance Codes

Some Known Hamming Distance Codes

k n |C|0 any 2n

n any 11 n=2r-1 2n/n+1

Perfect <n, k> Code (i.e., smallest possible) : 2n/B(n, k)

Hamming Codes

For any k: existence of n2n/B(n, k) => not Perfect Problem: no explicit construction

Cross Product Method (Explicit HD <n, k> Codes)Start with <n/t, k/t> code DLet C = D x D x … x D (t times)Claim: C is a <n, k> covering codeProof:

s = s1 s2 s3 … st

c = d1 d2 d3 … dt

≤k/t ≤k/t ≤k/t ≤k/tdist(s, c) ≤ k

Example of Cross Product Methodn = 10, k = 4, t=2 => use a <5, 2>

code D D = {00000, 11111}

00000--11111

11111--11111

11111--00000

00000--00000

1100011100≤2+2

1110000001

≤2+1=3

11000--11100

11100--00001

Size of Cross Product Codes: Dk

Assume D is perfect (e.g., Hamming code)

Perfect <n, k> code:

For large n, small t => same asymptotic size

Example: n, k=2, t=2

Outline

• Covering Code

2. Explicit Construction of Hamming Distance Covering

3. Explicit Construction of Edit Distance Covering Codes

Edit Distance Fuzzy Joins

cadb…

<abcd, eabc>

…<dddd, dadd>

Input Output

strings of length n over alphabet A (i.e.,|A|n strings)

Covering codes algorithm works in the same way: If C is a <n, k> edit distance code Send s to all code words at distance k+d/2

Differences with Hamming Distance

1. Length of code words might be different E.g. 1 insertion, |c| = n+1 => insertion-1 code E.g. 1 deletion, |c| = n-1 => deletion-1 code

2. Different code words might have different ball sizes

3. No known perfect codes or explicit construction

ababa…a(n+1)

aaba…a(n) abba…a

abaa…a(n)

baba…a(n)

… …

aaaaa…a(n+1)

aaaa..a(n)

Insertion-1 Codes

Let n=5, |A|=a=4, code words are of length 6Letters as integers from 0 to (a-1): e.g. 0230, 1124, …Let si be the ith digit of s1. sum(s) = 2. score(s) = sum(s) % (n+1)(a-1) (e.g., 6*3=18)3. R = Any a-1 consecutive residues:

e.g. {0,1,2}, {12,13,14}, {16,17,0}C = {003000, 303000, 003001, 003002, 200000, …}

**factor a worse than best possible**

Example: s=23010, sum(s)=24, score(s)=6

sum 23 24 25 26 27 28

38 39 40 41 42 43

score 5 6 7 8 9 10

0 1 2 3 4 5 6 7

X 023010 Y323010

sum 23 24 25 26 27 28

38 39 40 41 42 43

score 5 6 7 8 9 10

0 1 2 3 4 5 6 7

X 203010 Y233010

sum 23 24 25 26 27 28

38 39 40 41 42 43

score 5 6 7 8 9 10

0 1 2 3 4 5 6 7

X 230010 Y233010

sum 23 24 25 26 27 28

38 39 40 41 42 43

score 5 6 7 8 9 10

0 1 2 3 4 5 6 7

X 230010 Y230310

sum 23 24 25 26 27 28

38 39 40 41 42 43

score 5 6 7 8 9 10

0 1 2 3 4 5 6 7

X 230100 Y230130

Edit Distance Codes

Insertion/Deletion Size Explicit/Existence

Insertion-1 explicit

Deletion-1 explicit

Deletion-2 explicit

Deletion-1 existence

Summary

1. Fuzzy Joins for Hamming and Edit Distance in One-round MR

2. Anchor Points Algorithm Covering Codes Flexible parallelism Better communication cost than naive

3. Explicit construction of Hamming distance covering codes

4. Explicit Construction of Edit distance covering codes

Open Questions

Fuzzy Joins in MR Minimum communication for a given per-reducer

memory for 1 round MR algorithms? Know the answer for only Hamming Distance 1

How about multi-round MR algorithms? Covering Codes

Are there smaller codes? Can we construct smaller codes explicitly? What is the size of the smallest codes?

Related Work

Fuzzy Joins in MR Fuzzy Joins Using MapReduce, Afrati et. al., ICDE 2012 Document Similarity Self-Join with MapReduce, Baraglia et. al.,

ICDM 2010 Efficient Parallel Set-similarity Joins Using MapReduce, Vernica

et. al., SIGMOD 2010 Efficient Similarity Joins for Near Duplicate Detection, Xiao et.

al., WWW 2008Covering Codes

Covering codes, Gary Cohen On Asymmetric Coverings and Covering Numbers, Applegate

et. al., Comb. Designs 2003 Asymmetric Binary Covering Codes, Cooper et. al., Comb.

Theory 2002

Questions?

Foto Afrati — National Technical University of Athens Anish Das Sarma — ClearList Inc .

Documents