Date post: | 21-Dec-2015 |
Category: |
Documents |
View: | 231 times |
Download: | 0 times |
A Subsequence Matching withGaps-Range-Tolerances Framework:A Query-By-Humming Application
Alexios KotsifakosUniversity of Texas at Arlington, University of Athens (Greece)
Joint work with
Panagiotis Papapetrou (Aalto University, Finland)
Jaakko Hollmen (Aalto University, Finland)
Dimitrios Gunopulos (University of Athens, Greece)
2
Outline
Motivation
Related Work
Problem Setting
SMBGT: The proposed method
Experimental Evaluation
Summary and Conclusions
3
Introduction – MotivationSuppose you hear a song but cannot recall its name… One solution: Hum a short part of it and perform a
search on a music repository to find it or songs similar to it.
Query-By-Humming (QBH): Given a hummed query song search a music db for the K most similar songs.
Subsequence Matching: Given a query sequence and a db of sequences, identify the subsequence in db that best matches the query.
4
Subsequence Matching in a Database
database
query
What subsequence of any database sequence is the best match for Q ?
5
Converting music pieces to time series Each music piece is a sequence of
notes.
Notes consist of pitch (frequency) and duration.
Part of the Happy Birthday music score.
The 2D time series representation of the above music score.Pitch interval : distance between two pitches.
IOIR : duration ratio of two consecutive notes
8
QBH: A noisy application domain Slight or more serious errors due to instant or temporary key or tempo loss error-tolerant method.
Allow skipping a bounded number of elements in both query and target sequences due to serious humming errors.
Constrain the length of the matching subsequence.
Optional constraint to the min. number of matching elements.
9
Our contributions A subsequence matching framework that allows for
a constrained number of gaps on both query and target sequences,
variable tolerance levels in the matching (no probabilistic model),
a matching range that constrains the max. match length, bounding the min. # of matched elements.
SMBGT: Given a query Q and a target sequence X (with |Q| << |X|) finds the subsequence of X that best matches Q, considering the framework.
Extensive comparative evaluation on QBH of several DB-based methods and a probabilistic model-based method.
10
Outline
Motivation
Related Work
Problem Setting
SMBGT: The proposed method
Experimental Evaluation
Summary and Conclusions
11
Related Work DP-based methods
Whole sequence matching: DTW and variants (cDTW, EDR, ERP) robust to misalignments and warps, LCSS allows for gaps, edit distance, Iliopoulos et al., etc.
Subsequence matching: SPRING finds the subsequences of evolving numerical streams closest to a query, DTWc and DTWs account for local adjustments of tempo, Han et al. (2007) performed uniform segmentation and sliding windows for near exact matching where user selects segments’ length.
n-gram methods for near exact matching All account for 1D + fail to handle noise imposed by
users. Probabilistic-based methods (HMMs)
12
Outline
Motivation
Related Work
Problem Setting
SMBGT: The proposed method
Experimental Evaluation
Summary and Conclusions
13
Problem Setting: Representing Musical Pieces Expressing pitch
Absolute pitch: frequency of the note. Pitch interval: frequency difference between two adjacent notes.
Encoding duration Inter-Onset-Interval (IOI): difference in time onsets of two
adjacent notes. IOI Ratio (IOIR): ratio of IOIs of two adjacent notes. Log IOI Ratio (LogIOIR): logarithm of IOIR.
To save computational time we deal with note transitions:<pitch interval, IOIR>, <pitch interval, LogIOIR>
Performed quantizations on pitch interval with modulo 12, and on LogIOIR to the closest integer or closest value in [-2, 2].
14
Problem Setting: Definitions
X = {x1, …, xn} representing music piece of length |X|,
pair of real values for pitch and duration, DB={X1, …,
XN }music db,
X[ts : te]={xts , …, xte} subsequence of X (not always continuous), Q = {q1, …, qn }.
Definition 1 (Variable error-tolerant match):
,p rj j jx x x X
and match with variable -tolerance, ,if, for a set of
constraints { , }:
fi j i j
ff fp r
q Q x X q x
( ) ( ) and ( , ) ( , ).f p f r rp p i r r i ji f q i j f q x
15
Problem Setting: Definitions
1 1 2 2
i
Consider [ : ] and [ : ] of equal length. Let , the indices of
those elements in and that are included in the corresponding subsequences.
If , , , 1,...,| | , and πi i
Q X
i Q i X Q
Q ts te X ts te G G
Q X
q x G G i G
+1 i i+1 i
1 1 2 2
2 2
-π , -γ ,
then pair { [ : ], [ : ]} defines a common gapped subsequence of and .
The longest such subsequence with - is called ( , ).
Q ts te X ts te Q X
te ts r SMBGT Q X
Definition 2 (Common bounded-gapped subsequence):
SMBGT: error-tolerant matching denoted as ε-match.
16
Problem Setting:Example and Problem Formulation Example
α = 2, β = 1, r = 6, ε = 1 (absolute tol.) Q = {6, 3, 10, 5, 3, 2, 9}, X = {1, 1, 3, 4, 6, 9, 2, 3, 1} Q[2:6] with GQ = {2, 4, 5, 6} and X[3:8] with GX = {3, 4,
7, 8} Q = {6, 3, 10, 5, 3, 2, 9}X = {1, 1, 3, 4, 6, 9, 2, 3, 1}
Problem: Given a database DB with N sequences of arbitrary lengths, a query sequence Q, and positive integers δ and r, find set
of the top-K subsequences with{ [ : ] | }i s e iS X t t X DB
| ( , [ : ]) | .i s eSMBGT Q X t t
SMBGT(Q, X) = {Q[2:6], X[3:8]}
17
Outline
Motivation
Related Work
Problem Setting
SMBGT: The proposed method
Experimental Evaluation
Summary and Conclusions
18
SMGT SMGT: α = β = inf. Array a (|Q+1|)*(|X|+1). For all i in {1,…,|Q|}, j in {1,…,|X|}
Matrix s keeps in si,j for each ai,j the start point of its best alignment.
Online computation: two 1D arrays prev, cur, track scores and start points (prev.value, cur.value, prev.start, cur.start) of j-1 and j columns of a.
Best solution (SMGT) in best = (bestvalue, beststart, bestend) accounting for δ.
Check subsequences with length r (Reset()). Update values: Match ai,j = 1 Mismatch left cell can be inherited if length < r-1, top cell
always.
0, ,0
1, 1,
1, , 1
0, 0
1,if
max{ , },otherwise.
j i
fi j i j
i ji j i j
a a
a q xa
a a
19
SMBGT
Mismatch (propagation()): store the largest # of matched elements that can be propagated vertically or horizontally, not violating α and β.
Astart , Bstart : store the latest match positions in X and Q
Additional propagation check in Reset().
( ) (left propagation)
( 1) (top propagation)
prevstart
curstart
j A i
i B i
20
SMBGT: Example - InitializationQ = {0, -4, 1, 2, -2}, X = {0, 0, -4, 3,
0, 2, -3, 1}
α = 2, β = 1, δ = 3, r = |Q| = 5, ε = 0
X 0 0 -4 3 0 2 -3 1
Q 0 0 0 0 0 0 0 0 0
0 0
-4 0
1 0
2 0
-2 0
Alignment array α
21
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1
-4 1
1 0
2 0
-2 0
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1
-4 1
1 0
2 0
-2 0
s X
Q 0 0 -4 3 0 2 -3 1
0 1
-4 1
1 0
2 0
-2 0
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1
-4 1
1 0
2 0
-2 0
22
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1 1
-4 1 1
1 0 0
2 0 0
-2 0 0
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1 2
-4 1 2
1 0 0
2 0 0
-2 0 0
s X
Q 0 0 -4 3 0 2 -3 1
0 1 2
-4 1 2
1 0 0
2 0 0
-2 0 0
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1 1
-4 1 1
1 0 0
2 0 0
-2 0 0
23
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1
-4 1 1 2
1 0 0 2
2 0 0 0
-2 0 0 0
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2
-4 1 2 3
1 0 0 3
2 0 0 0
-2 0 0 0
s X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2
-4 1 2 2
1 0 0 2
2 0 0 0
-2 0 0 0
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1
-4 1 1 2
1 0 0 2
2 0 0 0
-2 0 0 0
24
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1
-4 1 1 2 2
1 0 0 2 2
2 0 0 0 0
-2 0 0 0 0
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2
-4 1 2 3 3
1 0 0 3 3
2 0 0 0 0
-2 0 0 0 0
s X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2
-4 1 2 2 2
1 0 0 2 2
2 0 0 0 0
-2 0 0 0 0
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1
-4 1 1 2 2
1 0 0 2 2
2 0 0 0 0
-2 0 0 0 0
25
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1
-4 1 1 2 2 2
1 0 0 2 2 2
2 0 0 0 0 0
-2 0 0 0 0 0
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5
-4 1 2 3 3 3
1 0 0 3 3 3
2 0 0 0 0 0
-2 0 0 0 0 0
s X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5
-4 1 2 2 2 2
1 0 0 2 2 2
2 0 0 0 0 0
-2 0 0 0 0 0
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1
-4 1 1 2 2 2
1 0 0 2 2 2
2 0 0 0 0 0
-2 0 0 0 0 0
26
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1
-4 1 1 2 2 2 1
1 0 0 2 2 2 0
2 0 0 0 0 0 3
-2 0 0 0 0 0 3
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5
-4 1 2 3 3 3 5
1 0 0 3 3 3 0
2 0 0 0 0 0 6
-2 0 0 0 0 0 6
s X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5
-4 1 2 2 2 2 5
1 0 0 2 2 2 0
2 0 0 0 0 0 2
-2 0 0 0 0 0 2
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1
-4 1 1 2 2 2 1
1 0 0 2 2 2 0
2 0 0 0 0 0 4
-2 0 0 0 0 0 4
27
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1
-4 1 1 2 2 2 1
1 0 0 2 2 2 0
2 0 0 0 0 0 3
-2 0 0 0 0 0 3
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5
-4 1 2 3 3 3 5
1 0 0 3 3 3 0
2 0 0 0 0 0 6
-2 0 0 0 0 0 6
s X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5
-4 1 2 2 2 2 5
1 0 0 2 2 2 0
2 0 0 0 0 0 2
-2 0 0 0 0 0 2
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1
-4 1 1 2 2 2 1
1 0 0 2 2 2 0
2 0 0 0 0 0 4
-2 0 0 0 0 0 4
28
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1
-4 1 1 2 2 2 1
1 0 0 2 2 2 0
2 0 0 0 0 0 1
-2 0 0 0 0 0 3
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5
-4 1 2 3 3 3 5
1 0 0 3 3 3 0
2 0 0 0 0 0 6
-2 0 0 0 0 0 6
s X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5
-4 1 2 2 2 2 5
1 0 0 2 2 2 0
2 0 0 0 0 0 6
-2 0 0 0 0 0 2
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1
-4 1 1 2 2 2 1
1 0 0 2 2 2 0
2 0 0 0 0 0 4
-2 0 0 0 0 0 4
29
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1
-4 1 1 2 2 2 1
1 0 0 2 2 2 0
2 0 0 0 0 0 1
-2 0 0 0 0 0 1
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5
-4 1 2 3 3 3 5
1 0 0 3 3 3 0
2 0 0 0 0 0 6
-2 0 0 0 0 0 6
s X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5
-4 1 2 2 2 2 5
1 0 0 2 2 2 0
2 0 0 0 0 0 6
-2 0 0 0 0 0 6
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1
-4 1 1 2 2 2 1
1 0 0 2 2 2 0
2 0 0 0 0 0 4
-2 0 0 0 0 0 4
30
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1 1
-4 1 1 2 2 2 1 1
1 0 0 2 2 2 0 0
2 0 0 0 0 0 1 1
-2 0 0 0 0 0 1 1
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5 5
-4 1 2 3 3 3 5 5
1 0 0 3 3 3 0 0
2 0 0 0 0 0 6 6
-2 0 0 0 0 0 6 6
s X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5 5
-4 1 2 2 2 2 5 5
1 0 0 2 2 2 0 0
2 0 0 0 0 0 6 6
-2 0 0 0 0 0 6 6
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1 1
-4 1 1 2 2 2 1 1
1 0 0 2 2 2 0 0
2 0 0 0 0 0 4 4
-2 0 0 0 0 0 4 4
31
SMBGT: Example - Computationa X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1 1 0
-4 1 1 2 2 2 1 1 0
1 0 0 2 2 2 0 0 2
2 0 0 0 0 0 1 1 2
-2 0 0 0 0 0 1 1 1
Astart X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5 5 0
-4 1 2 3 3 3 5 5 0
1 0 0 3 3 3 0 0 8
2 0 0 0 0 0 6 6 8
-2 0 0 0 0 0 6 6 6
s X
Q 0 0 -4 3 0 2 -3 1
0 1 2 2 2 5 5 5 0
-4 1 2 2 2 2 5 5 0
1 0 0 2 2 2 0 0 5
2 0 0 0 0 0 6 6 5
-2 0 0 0 0 0 6 6 6
Bstar
t
X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1 1 0
-4 1 1 2 2 2 1 1 0
1 0 0 2 2 2 0 0 3
2 0 0 0 0 0 4 4 3
-2 0 0 0 0 0 4 4 4
32
SMBGT: Example - Solution
a X
Q 0 0 -4 3 0 2 -3 1
0 1 1 1 1 1 1 1 0
-4 1 1 2 2 2 1 1 0
1 0 0 2 2 2 0 0 2
2 0 0 0 0 0 1 1 2
-2 0 0 0 0 0 1 1 1
Q = 0-4 1 2 -2
bestvalue = 3, beststart = 2, bestend = 6
X = 0 0 -4 3 0 2 -3 1skipped
33
Outline
Motivation
Related Work
Problem Setting
SMBGT: The proposed method
Experimental Evaluation
Summary and Conclusions
34
Experimental Setup: Data & Evaluation Music DB: 5643 freely available MIDI files.
6 synthetic query sets, 100 queries/set of lengths 13-137 Q0: exact segments of the db. Q.10 – Q.50: randomly modified 10-50% elements (both
dimensions) of each query in Q0, with at most 3 consecutive elements changed.
100 hummed queries of lengths 14-76.
DP and Model-based competitor methods DTW variants: DTWs, DTWc, SPRING modified to allow r. Edit: Version for music modified for LogIOIR + quantizations Iliopoulos et al.: Elastic version for subsequence matching
supporting constant and variable tolerance, modified to allow r. HMMs: Each db sequence modeled by an HMM.
35
Experimental Setup: Evaluation measures Recall: % of queries for which correct
answer is in top-K. MRR: mean inverse rank of queries in their
top-K results. AR: average rank of all queries in the set,
where rank is the # of sequences with score at least as good as that of the correct match.
36
Experimental Setup: Variable Tolerances Pitch
Absolute and relative tolerances tested, withand for relative tolerance two cases considered
Duration IOIR: LogIOIR:
( ) * , 0.2,0.25,0.5f pp ii q t t
/(1 ) *(1 ), , 0
/(1 ) *(1 ), , 0
p f p p f p pj p i j p i j
p f p p f p pj p i j p i j
x q x q x
x q x q x
( , ) { 2* , 0.5}f r r r rr j i j ii j x q x q
2 2
2 2
{0 log ( / ) 1}, log 0( , )
{| log ( / ) | 1}, log 0
r r rj i jf
r r r rj i j
x q xi j
x q x
37
Experimental Results: Synthetic Queries (1/4) Parameters
r = |Q|, δ = 0.9, 0.7. 0.6, 0.5, 0.35, 0.3 for Q0 - Q.50 , α = β = 3,tested both tolerance schemes, K = 20.
Best accuracy SMBGT: variable abs. tolerance, t = 0.2. SMGT: variable abs. tolerance, t = 0.25. Il. et al.: constant rel. tolerance with εp = 1, εr = 4. HMM: #states = 5, Gaussian observation distribution of the
states.
Representation ↑ noise: high recall for representations subset of lower
noise levels’ DP methods: <mod12, IOIR>, <pitch interval, IOIR> HMM: <mod12, LogIOIR in [-2,2]> smallest alphabet
38
Experimental Results: Synthetic Queries (2/4) Q0
DP-based methods: 100% recall, MRR = 1, AR = 1 for top-5.
DTWs: 96% recall for K = 250.
HMM: 96% recall for K = 5, MRR=0.95, AR = 1.22.
39
Experimental Results: Synthetic Queries (3/4) Q0
DP-based methods: 100% recall, MRR = 1, AR = 1 for top-5.
DTWs: 96% recall for K = 250.
HMM: 96% recall for K = 5, MRR=0.95, AR = 1.22.
41
Experimental Results: Hummed Queries Tolerance
Small constant and variable t better than greater values.
Variable t better than constant. Absolute better than relative. SMBGT and SMGT: absolute, t = 0.2 and 0.25.
Any tol. scheme.
42
Outline
Motivation
Related Work
Problem Setting
SMBGT: The proposed method
Experimental Evaluation
Summary and Conclusions
43
Summary
1. Lack of flexibility for existing subsequence matching methods. Edit most promising, but 30 times lower recall than SM(B)GT for K = 50, α, β improve accuracy.
2. Variable ε-tolerance better than constant tolerance.3. Small range for α, β tested. Other applications
cross validation.4. δ tuned according to hummers’ skills, r not exceed
1.2*|Q|.5. Simple representation favor DP-based methods.6. DP-based methods have similar retrieval time
complexity.
44
Conclusions and Future Work We proposed a novel subsequence matching
framework. We showed that it can be applied to noisy
application domains, such as QBH. We performed an extensive experimental
evaluation on a large collection of music songs using hummed queries.
Directions for future work include Testing the proposed method on other noisy domains. Developing a software application based on SMBGT. Indexing.
48
SMBGT
Theorem: SMBGT is not metric, as it does not follow the triangle inequality. Consider: 1D time series X1 = {3, 2}, X2 = {4}, X3 =
{2, 2, 1} ε-tolerance = 1, r = inf, δ = 0, α = β = 1 SMBGT(X1, X2) = 1, SMBGT(X2, X3) = 0,
SMBGT(X1, X3) = 2, butSMBGT(X1, X2) + SMBGT(X2, X3) < SMBGT(X1, X3).
50
DP Methods – Edit distance Latest version, extended for LogIOIR and
quantizations. PitchRange, DurationRange : max pitch interval, and
LogIOIR range in DB. Reports minj{α|Q|,j}.
51
DP Methods – SPRING
w(qi, xj): Lp of qi and xj
s to keep start points of best alignments. At each iteration, the start point of the element used to produce dbest is propagated.
Reports minj{α|Q|,j}.
52
DP Methods – DTW tempo scaling c: user-defined positive integer w(qip, xjp) = |qip - xjp| Report minj{α|Q|,j}.
53
DP Methods – Iliopoulos et al. Designed for whole sequence matching. Demands all points of Q match within constant ε-
tolerance in a subsequence of X. Allows limited # of gaps in X. Developed a more elastic version suitable for
subsequence matching Single db scan identifies all possible start points of
candidate matches. For each candidate, perform DP computation. Whole query matching requirement eliminated. When
non-matched query element is met, we consider Q[1:i] candidate match.
54
HMM Doubly stochastic process defined by
M distinct states L distinct symbols that can be observed at each state –
alphabet T = {tij} of transition probabilities, where tij = P[st = j|st-1 = i],
1<=i,j<=M E = {ej(k)} of the probabilities of observation symbols at j,
where ej(k) = P[ot = k|st = j] Π = {πj} of prior probabilities, where πj = P[s1 = j], 1<=j<=M.
Reflects the probabilistic relations of symbols within sequences.
Look for the model maximizing the log-likelihood of the query.
Model each db sequence with one HMM.
55
Data Database
Covers various music genres, e.g., blues, rock, classical, jazz, themes from movies and tv series, total of 40891 time series.
Converting music pieces to time series (offline process): All-channels extraction: for all channels, extracted the highest
pitch at every time click. Converted tuples <pitch, click> to <pitch interval, IOIR>.
Synthetic Queries Pitch interval change by ±k in [3, 8] (error by human + audio
tool). IOIR modified by ±k (real) in [2, 4]. At most 3 consecutive elements were replaced by noisy values.
Hummed Queries 4 males (2 with + 2 without musical training), 25 songs each. Conversion to MIDIs with Akoff. Several genres of music, no lyrics, much noise several versions
of each song.