Date post: | 20-Dec-2015 |
Category: |
Documents |
View: | 213 times |
Download: | 0 times |
Searching strings using the waves
An efficient index structure for string databases
Ingmar BrounsJacob Kleerekoper
Overview
Introduction A lower bound on the edit distance What is a wavelet? A refinement of the lowerbound MRS index Searching using the MRS index Questions
String searching
Searching in a string database S = {s1, …, sd}
Range searchFind all the substrings of S within a distance r
to a search string qThe error rate is denoted by ε = r / |q|
K-Nearest neighbor searchFind the k-closest substrings of S to q
Question from Adriano
What is the query range r in ε = r / |q|?
When performing range search, r denotes the maximum value the edit distance of a substring of the database may have to be a valid result
f(s) is a frequency vector
Σ is an alphabet of σ characters s is a string from that alphabet Σ f(s) = [v1, …, vσ]
where vi = frequency of i-th letter of Σ in s
Sum of v1, …, vσ = length of s
Example of f(s)
Σ = {A, C, G, T} s = CTACATCGATCGATCAG #A = 5, #C = 5, #G = 3, #T = 4 f(s) = [5, 5, 3, 4] Sum of v1, …, vσ = length of s
5 + 5 + 3 + 4 = 17
Question from Laurence
“As a result of lemma 1, the transformation of a string of length n lie on the σ-1 dimensional plane that passes through the point [n, 0, …, 0] and is perpendicular to the normal vector [1, …, 1]”Why is that the case?
Answer to Laurence Take a string s with |s| = 4 and an alphabet Σ = {A, B, C} This string might be AAAA, BBBB, CCCC and 21 more The corresponding f(s) are [4, 0, 0], [0, 4, 0] and [0, 0, 4].
All the f(s) span the same 2D plane in the 3D space with equation v1 + v2 + v3 = 4 which makes the normal vector [1, 1, 1]
This holds in general for every alphabet length n since v1
max + … + vn max = n has always the normal vector [1, …, 1]
f(s) = [v1, ..., vσ]
Insert
vi := v
i + 1
Delete
vi := v
i - 1
Replace
vi := v
i + 1 and v
j := v
j - 1
with i ≠ j
Edit operations on s
The space in which all the possible points [v1, ..., vσ] exist
Take u and v points in the σ-dimensional space
We call u and v neighbors if you can obtain u from v using one edit operation
The σ-dimensional space
Take u and v frequency vectors of two strings of the same alphabet (points in the σ-dimensional space)
The Frequency distance FD1 (u, v)
between u and v is the minimum number of steps to get from u to v by jumping each step to a neighbor point
Frequency distance FD1
The edit distance ED(s1, s
2) of strings s
1 and
s2 is the minimal number of edit operations to
get from s1 to s
2
FD1(f(s
1), f(s
2)) ≤ ED (s
1, s
2)
s1 = AC, s
2 = CA, f(s
1) = [1,1,0,0], f(s
2) =
[1,1,0,0]
FD1 = 0
ED = 2 (two replaces or one insert and one delete)
Frequency distance vs. edit distance
FD1(f(s
1), f(s
2)) ≤ ED (s
1, s
2)
Proof:
In case of a single insert or delete in ED, rule 1 or 2 is used in FD
1. Now ED as well as FD
1 are
incremented resp. decremented by 1
In case of an insert and a delete or a replace in the ED, the FD
1 always uses rule 3: v
i := v
i + 1
and vj := v
j – 1
This will result in a lower value for the FD1 than
the ED, hence the ≤ sign
So FD1 is the lower bound on ED
Frequency distance vs. edit distance
Take q and s strings from alphabet Σ, r is the range (maximum ED in range search)
if r < FD1(f(q), f(s))
then r < ED(q, s)
To compute ED costs O(nm) time, but FD
1 costs only O(σ)
The lower bound of ED
Take frequency vectors u and v of two strings of the same alphabet Σ
We collect total positive distance (pos) and total negative distance (neg)
for every letter i in σ
if ui > v
i we add the difference u
i – v
i to pos
otherwise we add vi – u
i to neg
return the maximum of pos and neg
Computing FD1
How does it work?
u1 < v
1, so add 8-2 to neg
u2 > v
2, so add 10-1 to
pos Now pos = 6, neg = 9 return 9
u = [2, 10]
v = [8, 1]
6 replaces and 3 inserts
Improving the lower bound
We’ve established a lower bound on the edit distance, namely de frequency distance
But we can improve this lower bound by incorpotating more information then how often letters occure. We would like to have more info about when they occur.
Wavelets
Wavelet transformProblems with Fourier transform
Representation of frequencies in signal But we do not know when these frequencies occur
Shows time & frequencies Used in all sorts of signal processing
(compression) JPEG2000
How does this affect us?Suppose we have some signalWe can encode this signal by recursively
taking the average of a part of the signal, and then the difference between the averages of half of this part.
Wavelets
Now with strings AT
frequency vector (average) = [1,0,0,1]Detail = [1,0,0,0] – [0,0,0,1] = [1,0,0,-1]Note that we know by the detail that the first
half was an A and the second half was a T.
Wavelets (Adriano)TCACTTAG
TCAC TTAG
TC AC TT AG
T C A C T T A G
[0,
0,
0,
1]
[0,
1,
0,
0]
[1,
0,
0,
0]
[0,
1,
0,
0]
[0,
0,
0,
1]
[0,
0,
0,
1]
[0,
0,
1,
0]
[0,
0,
1,
0]
WaveletsTCACTTAG
TCAC TTAG
TC AC TT AG
T C A C T T A G
[0,
0,
0,
1]
[0,
1,
0,
0]
[1,
0,
0,
0]
[0,
1,
0,
0]
[0,
0,
0,
1]
[0,
0,
0,
1]
[1,
0,
0,
0]
[0,
0,
1,
0]
[0,
1,
0,
1]
[1,
1,
0,
0]
[0,
0,
0,
2]
[1,
0,
1,
0]
TCACTTAG
TCAC TTAG
TC AC TT AG
T C A C T T A G
[0,
0,
0,
1]
[0,
1,
0,
0]
[1,
0,
0,
0]
[0,
1,
0,
0]
[0,
0,
0,
1]
[0,
0,
0,
1]
[1,
0,
0,
0]
[0,
0,
1,
0]
[0,
1,
0,
1]
[1,
1,
0,
0]
[0,
0,
0,
2]
[1,
0,
1,
0]
[1,
2,
0,
1]
[1,
0,
1,
2]
[2,
2,
1,
3]
TCACTTAG
TCAC TTAG
TC AC TT AG
T C A C T T A G
[0,
0,
0,
1]
[0,
1,
0,
0]
[1,
0,
0,
0]
[0,
1,
0,
0]
[0,
0,
0,
1]
[0,
0,
0,
1]
[1,
0,
0,
0]
[0,
0,
1,
0]
[0,
1,
0,
1]
[1,
1,
0,
0]
[0,
0,
0,
2]
[1,
0,
1,
0]
[1,
2,
0,
1]
[1,
0,
1,
2]
[2,
2,
1,
3]
[0,
-1,
0,
1]
TCACTTAG
TCAC TTAG
TC AC TT AG
T C A C T T A G
[0,
0,
0,
1]
[0,
1,
0,
0]
[1,
0,
0,
0]
[0,
1,
0,
0]
[0,
0,
0,
1]
[0,
0,
0,
1]
[1,
0,
0,
0]
[0,
0,
1,
0]
[0,
1,
0,
1]
[1,
1,
0,
0]
[0,
0,
0,
2]
[1,
0,
1,
0]
[1,
2,
0,
1]
[1,
0,
1,
2]
[2,
2,
1,
3]
[0,
2,
-1,
-1]
[-1,
0,
0,
1]
[0,
-1,
0,
1]
[1,
-1,
0,
0]
[1,
-1,
0,
0]
[-1,
0,
-1,
2]
[1,
0,
-1,
0]
TCACTTAG
TCAC TTAG
TC AC TT AG
T C A C T T A G
[2,
2,
1,
3]
[0,
2,
-1,
-1]
[-1,
0,
0,
1]
[0,
-1,
1,
0]
[1,
-1,
0,
0]
[1,
-1,
0,
0]
[-1,
0,
-1,
2]
[1,
0,
-1,
0]
The kth wavelet transformation
Definition 4 Let s=c0,...cn-1 be a string from the alphabet {α1,.., ασ}, then kth-level wavelet transformation, ψk(s), 0 ≤ k ≤ log2n, of s is defined as:
ψk(s) = [vk,0,..,vk,n/(2^k)-1] where vk,i = [Ak,i,Bk,i]
The 0th wavelet transformation
The 0th wavelet transformation defines the original string
ψk(s) = [v0,0,..,v0,(n/1)-1] where vk,i = [Ak,i,Bk,i] For TCACTTAG that is V0,0,..,V0,7
A0,0,..A0,7
The log2n wavelet transformation
In the article they only chose to use the first and second wavelet coefficient, this corresponds to the log2n wavelet transformation.
ψk(s) = [vk,0,..,vk,n/(2^k)-1] so only vlog(n),0 For TCACTTAG that is v3,0 A3,0= A2,0+A2,1 A2,0 = A1,0 + A1,1 , A2,1 = A1,2 + A1,3 A1,0 = A0,0 + A0,1 , A1,1 = A0,2 + A0,3 etc A3,0=[2,2,1,3] B3,0 = A2,0 – A2,1 = [0,2,-1,-1]
Theorem 3 (Bogdan)
String S with coefficients [A,B] Where A=[a1,..,aσ] and B = [b1,..,bσ] How can an edit operation influence A and B Replace first half & second half
ai:= ai+1, aj:= aj-1, bi:= bi+1 , bj:= bj-1
ai:= ai+1, aj:= aj-1, bi:= bi-1 , bj:= bj+1
Delete & Insert ai:= ai+-1 , bj:= bj+-1
Theorem 3
Delete on string of even lengthai:= ai+-1 , bi:= bi-1 , bj:= bj+2AABA , A=[3,1] B=[1,-1]ABA A=[2,1] B=[0,1]
Insert on string of odd lengthai:= ai+-1 , bi:= bi+1 , bj:= bj-2ABA A=[2,1] B=[0,1]AABA , A=[3,1] B=[1,-1]
The lower bound
So if we have ψ(si) and ψ(sj), the five steps listed at theorem two can be used to walk from ψ(si) to ψ(sj). (These are two points in 2σ dimensional
space) So now the FD2(ψ(si), ψ(sj)) is the shortest legal
path using these steps from ψ(si) to ψ(sj)
A table of trees Ti,j with index structure
A column stands for string sj of database S =
{s1, ..., s
d} with 1 < j < d
A row stands for a resolution (or window-size) 2i with a < i < a + l -1 and l the number of resolution-levels in the index
Each tree Ti,j consists of several Minimum
Bounding Rectangles, MBR's, containing several wavelet-coefficients depending on the given capacity c of the MBR
The MRS index structure
Take string s1 = CTAGTCGA
Let's build the tree T2,1
, given c = 3
window-size w = 4 (= 2i)string-number j = 1
Take a window of size w and slide along s1
The first MBR contains the 1st and 2nd coefficient of the first c substrings in the window:{φ(CTAG), φ(TAGT), φ(AGTC)}next MBR: {φ(GTCG), φ(TCGA)}
Building the MRS index (1)
s1 = CTAGTCGA, c = 3
So T2,1
= {φ(CTAG), φ(TAGT), φ(AGTC)},
{φ(GTCG), φ(TCGA)}
Next T3,1
= {φ(CTAGTCGA)} and consist of
only 1 MBR
Normally sj is much bigger than a + l - 1
(the maximum resolution)
Et cetera
Building the MRS index (2)
Search for a subquery of length 2i?Just take row R
i = {T
i,1, ..., T
i, d} of the table
Take a query string q and a MBR B:FD(q, B) is the minimum of all the FD(q, s) where s Є B, soif r ≤ FD(q, B) then r ≤ FD(q, s) for all s Є B
Wavelet coefficients of substrings obtained by sliding the window are very close to each other, so the set of coefficients in an MBR are highly clustered.
Some remarks on the index structure
Range Queries
We are searching for all sequences that are within an edit distance of r from the query string
Easy case: the index contains a resolution that exactly fits the size of the query string
Range queries
Query string has length 2a
For the corresponding row in the index, for every database sequence, we compute the FD of the query string to the MBR’s.
If r ≤ FD(q,B) then r ≤ FD(q,s) for all s elem B If r < FD(q,B) then r < ED(q,s) for every s elem B So if FD(q,B) > r , then we drop B
Range queries
However we may have some false positives. If r > FD(q,s) this does not guarantee that
r > ED(q,s) for every s elem B Thats why we have to post process all
strings that are in the boxes for which r > FD(q,B). (e.g. By dynamic programming)
Range queries
Now what if there is no row with a resolution corresponding to the query string.
We partition the query stringWe take the longest possible suffix such that
the resolution exists in the indexWe continue doing this interatively, so we get
q1,q2,..,qt
Range queries
Nearest neighbour queries
Given some query we search for the k closest substrings in the database.
Phase 1 Lookup the set of k closest MBRs to the query r1 is the kth smallest edit distance to strings in the set
Phase 2 RangeSearch(q,r1) Return the k closest strings
Why phase 2 ? FDbox10 ≤ FDbox11,FD10 ≤ ED10, FD11 ≤ ED11
However this does not guarantee that ED10 ≤ ED11
Questions (Peter)
It is nice that they can prove that the MRS index does not incur any false drops (Theorem 4), but is this also true in a practical sense? If r ≤ FD(q,B) then r ≤ FD(q,s) for all s elem B If r < FD(q,B) then r < ED(q,s) for every s
elem BSo if FD(q,B) > r , then we drop B
Questions (Lee)
The article focuses on substring matching. What adaptations would we need for whole matching?
Determine [A,B] for every entire string in the DB
Determine FD of q to each string
Questions (Bogdan)
The definition of FD(q,B) in section 3.3 says that the distance between a query transformation and a box is the minimum of the distances between the query transformation and the transformations in that box. It is also mentioned in the same section that for each box (MBR) only the lower/higher end points and the starting location of the first substring contained in that MBR are stored as part of the index
Further on, in section 4.4, a part of the range query algorithm implies the computation of FD(q,B) for various (query, MBR) pairs. However, since we only have the lower/higher end points for each MBR, how is it possible to compute FD(q,B) without retrieving all the substrings s_i that are in the box B from the disk
I could think of alternatively defining FD(q,B) with a formula involving only the lower/higher end points of the box B, but this is not what the authors are suggesting/using.
Questions(Bogdan)
I read the article several times, and I don't understand wavelet coefficients.What does this mean and how can it be used for string comparison?
Do the wavelet coefficients depend on the data itself or only on the frequency of the appearance of the data?
Probably clear by Ingmars explanation
Questions from Marjolijn
In the article is said that they use the edit distance. But they also mention the weighted edit distance (ED), why don't they use this one? Does it take more calculation time?
In the FD you don’t know anymore if there was an delete + insert or a replace.
Questions from Marjolijn
Is the algorithm with the edit distance useful for data like DNA where we know for sure that several changes depend on each other and occur more often than other ones?
Same answer as previous slideNo, not if you take these special occasions into account
Questions from Marjolijn