12th International Society for Music Information Retrieval Conference (ISMIR 2011) SCORE-INFORMED VOICE SEPARATION FOR PIANO RECORDINGS Sebastian Ewert Computer Science III, University of Bonn [email protected]Meinard M ¨ uller Saarland University and MPI Informatik [email protected]ABSTRACT The decomposition of a monaural audio recording into mu- sically meaningful sound sources or voices constitutes a fun- damental problem in music information retrieval. In this paper, we consider the task of separating a monaural piano recording into two sound sources (or voices) that correspond to the left hand and the right hand. Since in this scenario the two sources share many physical properties, sound sep- aration approaches identifying sources based on their spec- tral envelope are hardly applicable. Instead, we propose a score-informed approach, where explicit note events speci- fied by the score are used to parameterize the spectrogram of a given piano recording. This parameterization then al- lows for constructing two spectrograms considering only the notes of the left hand and the right hand, respectively. Fi- nally, inversion of the two spectrograms yields the separa- tion result. First experiments show that our approach, which involves high-resolution music synchronization and para- metric modeling techniques, yields good results for real- world non-synthetic piano recordings. 1. INTRODUCTION In recent years, techniques for the separation of musically meaningful sound sources from monaural music recordings have been applied to support many tasks in music infor- mation retrieval. For example, by extracting the singing voice, the bassline, or drum and instrument tracks, signif- icant improvements have been reported for tasks such as in- strument recognition [7], melody estimation [1], harmonic analysis [10], or instrument equalization [9]. For the sepa- ration, most approaches exploit specific spectral or tempo- ral characteristics of the respective sound sources, for ex- ample the broadband energy distribution of percussive ele- ments [10] or the spectral properties unique to the human vocal tract [1]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. c 2011 International Society for Music Information Retrieval. 0 2 4 6 8 10 C3 E3 G#3 C4 E4 G#4 C5 E5 G#5 0 2 4 6 8 10 12 14 16 0 2 4 6 8 10 12 14 16 0 2 4 6 8 10 12 14 16 Figure 1. Decomposition of a piano recording into two sound sources corresponding to the left and right hand as specified by a musical score. Shown are the first four measures of Chopin Op. 28 No. 15. In this paper, we present an automated approach for the decomposition of a monaural piano recording into sound sources corresponding to the left and the right hand as speci- fied by a score, see Figure 1. Played on the same instrument and often being interleaved, the two sources share many spectral properties. As a consequence, techniques that rely on statistical differences between the sound sources are not directly applicable. To make the separation process feasi- ble, we exploit the fact that a musical score is available for many pieces. We then use the explicitly given note events of the score to approximate the spectrogram of the given piano recording using a parametric model. Characterizing which part of the spectrogram belongs to a given note event, the model is then employed to decompose the spectrogram into parts related to the left hand and to the right hand. As an application, our goal is to extend the idea of an instru- ment equalizer as presented in [9] to a voice equalizer that can not only emphasize or attenuate whole instrument tracks but also individual voices or even single notes played by the same instrument. While we restrict the task in this paper to the left/right hand scenario, our approach is sufficiently general to isolate any kind of voice (or group of notes) that is specified by a given score. So far, score-informed sound separation has received 245
Transcript
12th International Society for Music Information Retrieval Conference (ISMIR 2011)
SCORE-INFORMED VOICE SEPARATION FOR PIANO RECORDINGS
Sebastian EwertComputer Science III, University of Bonn
The decomposition of a monaural audio recording into mu-sically meaningful sound sources or voices constitutes a fun-damental problem in music information retrieval. In thispaper, we consider the task of separating a monaural pianorecording into two sound sources (or voices) that correspondto the left hand and the right hand. Since in this scenariothe two sources share many physical properties, sound sep-aration approaches identifying sources based on their spec-tral envelope are hardly applicable. Instead, we propose ascore-informed approach, where explicit note events speci-fied by the score are used to parameterize the spectrogramof a given piano recording. This parameterization then al-lows for constructing two spectrograms considering only thenotes of the left hand and the right hand, respectively. Fi-nally, inversion of the two spectrograms yields the separa-tion result. First experiments show that our approach, whichinvolves high-resolution music synchronization and para-metric modeling techniques, yields good results for real-world non-synthetic piano recordings.
1. INTRODUCTION
In recent years, techniques for the separation of musicallymeaningful sound sources from monaural music recordingshave been applied to support many tasks in music infor-mation retrieval. For example, by extracting the singingvoice, the bassline, or drum and instrument tracks, signif-icant improvements have been reported for tasks such as in-strument recognition [7], melody estimation [1], harmonicanalysis [10], or instrument equalization [9]. For the sepa-ration, most approaches exploit specific spectral or tempo-ral characteristics of the respective sound sources, for ex-ample the broadband energy distribution of percussive ele-ments [10] or the spectral properties unique to the humanvocal tract [1].
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page.
Figure 1. Decomposition of a piano recording into two soundsources corresponding to the left and right hand as specified by amusical score. Shown are the first four measures of Chopin Op. 28No. 15.
In this paper, we present an automated approach for thedecomposition of a monaural piano recording into soundsources corresponding to the left and the right hand as speci-fied by a score, see Figure 1. Played on the same instrumentand often being interleaved, the two sources share manyspectral properties. As a consequence, techniques that relyon statistical differences between the sound sources are notdirectly applicable. To make the separation process feasi-ble, we exploit the fact that a musical score is available formany pieces. We then use the explicitly given note eventsof the score to approximate the spectrogram of the givenpiano recording using a parametric model. Characterizingwhich part of the spectrogram belongs to a given note event,the model is then employed to decompose the spectrograminto parts related to the left hand and to the right hand. Asan application, our goal is to extend the idea of an instru-ment equalizer as presented in [9] to a voice equalizer thatcan not only emphasize or attenuate whole instrument tracksbut also individual voices or even single notes played by thesame instrument. While we restrict the task in this paperto the left/right hand scenario, our approach is sufficientlygeneral to isolate any kind of voice (or group of notes) thatis specified by a given score.
So far, score-informed sound separation has received
245
Poster Session 2
only little attention in the literature. In [11], the authorsreplace the pitch estimation step of a sound separation sys-tem for stereo recordings with pitch information providedby an aligned MIDI file. In [6], a score-informed sys-tem for the elimination of the solo instrument from poly-phonic audio recordings is presented. For the descriptionof the spectral envelope of an instrument, the approach re-lies on pretrained information from a monophonic instru-ment database. In [4], score information is used as priorinformation in a separation system based on probabilisticlatent component analysis (PLCA). This approach is in [8]compared to a score-informed approach based on paramet-ric atoms. In [9], a score-informed system for the extractionof individual instrument tracks is proposed. To counterbal-ance their harmonic and inharmonic submodels, the authorshave to incorporate complex regulation terms into their ap-proach. Furthermore, the authors presuppose that, for eachaudio recording, a perfectly aligned MIDI file is available,which is not a realistic assumption.
In this paper, our main contribution is to extend the ideaof an instrument equalizer to a voice equalizer that doesnot rely on statistical properties of the sound sources. Asa further contribution, we do no presuppose the existence ofprealigned MIDI files. Instead, we revert to high-resolutionmusic synchronization techniques [3] to automatically alignan audio recording to a corresponding musical score. Usingthe aligned score as an initialization, we follow the paramet-ric model paradigm [2, 6, 7, 9] to obtain a note-wise param-eterization of the spectrogram. As another contribution weshow how separation masks that allow for a construction ofvoice-specific spectrograms can be derived from our model.Finally, applying a Griffin-Lim based inversion [5] to theseparated spectrograms yields the final separation result.
The remainder of this paper is organized as follows. InSection 2, we introduce our parametric spectrogram model.Then, in Section 3, we describe how our model is employedto decompose a piano recording into two voices that cor-respond to the left hand and the right hand. In Section 4,we report on our systematic experiments using real-worldas well as synthetic piano recordings. Conclusions andprospects on future work are given in Section 5. Furtherrelated work is discussed in the respective sections.
2. PARAMETRIC MODEL
To describe an audio recording of a piece of music usinga parametric model, one has to consider many musical andacoustical aspects [7, 9]. For example, parameters are re-quired to encode the pitch as well as the onset position andduration of note events. Further parameters might encodetuning aspects, the timbre of specific instruments, or ampli-tude progressions. In this section, we describe our modeland show how its parameters can be estimated by an itera-tive method.
1 2 3 40
500
1000
1500
0
2
4
6
8
10
12
14
16
18
20
(a)
0 0.5 1 1.5 2 2.50
500
1000
1500
0
2
4
6
8
10
12
14
16
18
20
(b)
1 2 3 40
500
1000
1500
0
2
4
6
8
10
12
14
16
18
20
(c)
1 2 3 40
500
1000
1500
0
2
4
6
8
10
12
14
16
18
20
(d)
1 2 3 40
500
1000
1500
0
2
4
6
8
10
12
14
16
18
20
(e)
Figure 2. Illustration of the first iteration of our parameter estima-tion procedure continuing the example shown in Figure 1 (shownsection corresponds to the first measure).(a): Audio spectrogramY to be approximated.(b)-(e) Model spectrogramYλ after cer-tain parameters are estimated.(b): ParameterS is initialized withMIDI note events.(c): Note events inS are synchronized with theaudio recording.(d): Activity α and tuning parameterτ are esti-mated.(e): Partials’ energy distribution parameterγ is estimated.
2.1 Parametric Spectrogram Model
Let X ∈ CK×N denote the spectrogram andY = |X| the
magnitude spectrogram of a given music recording. Fur-thermore, letS := {µs | s ∈ [1 : S]} denote a set ofnote events as specified by a MIDI file representing a mu-sical score. Here, each note event is modelled as a tripleµs = (ps, ts, ds), with ps encoding the MIDI pitch,ts theonset position andds the duration of the note event. Ourstrategy is to approximateY by means of a model spectro-gramY S
λ , whereλ denotes a set of free parameters repre-senting acoustical properties of the note events. Based onthe note event setS, the model spectrogramY S
λ will be con-structed as a superposition of note-event spectrogramsY s
λ ,s ∈ [1 : S]. More precisely, we defineY S
λ at frequency bink ∈ [1 : K] and time framen ∈ [1 : N ] as
Y Sλ (k, n) :=
∑
µs∈S
Y sλ (k, n), (1)
246
12th International Society for Music Information Retrieval Conference (ISMIR 2011)
where eachY sλ denotes the part ofY S
λ that is attributed toµs. EachY s
λ consists of a component describing the ampli-tude or activity over time and a component describing thespectral envelope of a note event. More precisely, we define
Y sλ (k, n) := αs(n) · ϕτ,γ(ωk, ps), (2)
whereωk denotes the frequency in Hertz associated with thek-th frequency bin. Furthermore,αs ∈ R
N≥0 encodes the ac-
tivity of the s-th note event. Here, we setαs(n) := 0, if thetime position associated with framen lies inR\ [ts, ts+ds].The spectral envelope associated with a note event is de-scribed using a functionϕτ,γ : R × [1 : P ] → R≥0, where[1 : P ] with P =127 denotes the set of MIDI pitches. Moreprecisely, to describe the frequency and energy distributionof the first L partials of a specific note event with MIDIpitch p ∈ [1 : P ], the functionϕτ,γ depends on a parame-ter τ ∈ [−0.5, 0.5]P related to the tuning and a parameterγ ∈ [0, 1]L×P related to the energy distribution over theL
partials. We define for a frequencyω given in Hertz the en-velope function
ϕτ,γ(ω, p) :=∑
ℓ∈[1:L]
γℓ,p · κ(ω − ℓ · f(p + τp)), (3)
where the functionκ : R→ R≥0 is a suitably chosen Gaus-sian centered at zero, which is used to describe the shape ofa partial in frequency direction, see Figure 3. Furthermore,f : R→ R≥0 defined byf(p) := 2(p−69)/12 · 440 maps thepitch to the frequency scale. To account for non-standardtunings, we use the parameterτp to shift the fundamentalfrequency upwards or downwards by up to half a semitone.Finally, λ := (α, τ, γ) denotes the set of free parameterswith α := {αs | s ∈ [1 : S]}. The number of free param-eters is kept low since the parametersτ andγ only dependon the pitch but not on the individual note events given byS. Here, a low number allows for an efficient parameter es-timation process as described below. Furthermore, sharingthe parameters across the note events prevents model over-fitting.
Now, finding a meaningful parameterization ofY can beformulated as the following optimization task:
λ∗ = argminλ
‖Y − Y Sλ ‖F , (4)
where‖·‖F denotes the Frobenius norm. In the following,we illustrate the individual steps in our parameter estima-tion procedure in Figure 2, where a given audio spectrogram(Figure 2a) is approximated by our model (Figure 2b-2e).
2.2 Initialization and Adaption of Note TimingParameters
To initialize our model, we exploit the available MIDI in-formation represented byS. For thes-th note eventµs =
0 500 1000 1500 2000 25000
0.2
0.4
0.6
0.8
1
ϕτ,γ(ω
,p)
Frequency in Hzℓ=1 ℓ=2 ℓ=3 ℓ=4 ℓ=5 ℓ=6 ℓ=7 ℓ=8 ℓ=9
γ1,p
γ2,p
γ3,p
γ4,pγ5,p
γ6,p γ7,p γ8,p γ9,p
Figure 3. Illustration of the spectral envelope functionϕτ,γ(ω, p)for p = 60 (middle C),τ = 0 and some example values for pa-rametersγ.
(ps, ts, ds), we setαs(n) := 1 if the time position associ-ated with framen lies in [ts, ts + ds] andαs(n) := 0 other-wise. Furthermore, we setτp := 0, γ1,p := 1 andγℓ,p := 0for p ∈ [1 : P ], ℓ ∈ [2 : L]. An example model spectrogramY S
λ after the initialization is given in Figure 2b.Next, we need to adapt and refine the model parameters
to approximate the given audio spectrogram as accuratelyas possible. This parameter adaption is simplified when theMIDI file is assumed to be perfectly aligned to the audiorecording as in [9]. However, in most practical scenariossuch a MIDI file is not available. Therefore, in our ap-proach, we employ a high resolution music synchronizationapproach as described in [3] to adapt the onset positionsof the note events setS. Based on Dynamic Time Warp-ing (DTW) and chroma features, the approach also incor-porates onset-based features to yield a high alignment accu-racy. Using the resulting alignment, we determine for eachnote event the corresponding position in the audio record-ing and update the onset positions and durations inS ac-cordingly. After the synchronization, the note event setSremains unchanged during all further parameter estimationsteps. Figure 2c shows an example model spectrogram afterthe synchronization step.
2.3 Estimation of Model Parameters
To estimate the parameters inλ, we look for (α, τ, γ) thatminimize the functiond(α, τ, γ) := ‖Y − Y S
(α,τ,γ)‖F , thusminimizing the distance between the audio and the modelspectrogram. Additionally, we need to consider range con-straints for the parameters. For example,τ is required to bean element of[−0.5, 0.5]P . To approximatively solve thisconstraint optimization problem, we employ a slightly mod-ified version of approach exerted in [2]. In summary, thismethod works iteratively by fixing two parameters and byminimizingd with regard to the third one using a trust regionbased interior-points approach. For example, to get a betterestimate forα, we fix τ andγ and minimized(·, τ, γ). Thisprocess is repeated until convergence similar to the well-known expectation-maximization algorithm. Figures 2d and2e illustrate the first iteration of our parameter estimation.Here, Figure 2d shows the model spectrogramY S
λ after theestimation of the tuning parameterτ and the activity param-
247
Poster Session 2
0 5 10 150
500
1000
1500
0
5
10
15
20
(a)
0 5 10 150
500
1000
1500
0
5
10
15
20
0 5 10 150
500
1000
1500
0
5
10
15
20
(b)
0 5 10 150
500
1000
1500
0
0.2
0.4
0.6
0.8
1
0 5 10 150
500
1000
1500
0
0.2
0.4
0.6
0.8
1
(c)
0 5 10 150
500
1000
1500
0
5
10
15
20
0 5 10 150
500
1000
1500
0
5
10
15
20
(d)
0 5 10 15
0 5 10 15
(e)
Figure 4. Illustration of our voice separation process continuing the example shownin Figure 1. (a) Model spectrogramY S
λ after theparameter estimation.(b) Derived model spectrogramsY L
λ andY R
λ corresponding to the notes of the left and the right hand.(c) SeparationmasksML andMR. (d) Estimated magnitude spectrogramsY L andY R. (e)Reconstructed audio signalsxL andxR.
eterα. Figure 2e showsY Sλ after the estimation of the par-
tials’ energy distribution parameterγ.
3. VOICE SEPARATION
After the parameter estimation,Y Sλ yields a note-wise para-
metric approximation ofY . In a next step, we employ infor-mation derived from the model to decompose the originalaudio spectrogram into separate channels or voices. To thisend, we exploit thatY S
λ is a compound of note-event spec-trogramsY s
λ . With T ⊂ S, we defineY Tλ as
Y Tλ (k, n) :=
∑
µs∈T
Y sλ (k, n). (5)
ThenY Tλ approximates the part ofY that can be attributed
to the note events inT . One way to yield an audible separa-tion result could be to apply a spectrogram inversion directlyto Y T
λ . However, to yield an overall robust approximationresult our model does not attempt to capture every possi-ble spectral nuance inY . Therefore, an audio recordingdeduced directly fromY T
λ would miss these nuances andwould consequently sound rather unnatural. Instead, we re-vert to the original spectrogram again and useY T
λ only toextract suitable parts ofY . To this end, we derive asepa-ration mask MT ∈ [0, 1]K×N from the model which en-codes how strongly each entry inY should be attributed toT . More precisely, we define
MT :=Y T
λ
Y Sλ + ε
, (6)
where the division is understood entrywise. The small con-stantε > 0 is used to avoid a potential division by zero.Furthermore,ε prevents that relatively small values inY T
λ
lead to large masking values, which would not be justifiedby the model. For our experiments, we setε = 10−2.
For the separation, we applyMT to a magnitude spec-trogram via
Y T := MT ◦ Y, (7)
where◦ denotes entrywise multiplication (Hadamard prod-uct). The resultingY T is referred to asestimated magnitudespectrogram. Here, using a mask for the separation allowsfor preserving most spectral nuances of the original audio.In a final step, we apply a spectrogram inversion to yield anaudible separation result. Here, a commonly used approachis to combineY T with the phase information of the origi-nal spectrogramX in a first step. Then, an inverse FFT incombination with an overlap-add technique is applied to theresulting spectrogram [7]. However, this usually leads toclicking and ringing artifacts in the resulting audio record-ing. Therefore, we apply a spectrogram inversion approachoriginally proposed by Griffin and Lim in [5]. The methodattenuates the inversion artifacts by iteratively modifying theoriginal phase information. The resultingxT constitutes ourfinal separation result referred to asreconstructed audio sig-nal (relative to T ).
Next, we transfer these techniques to our left/right handscenario. Each step of the full separation process is illus-trated by Figure 4. Firstly, we assume that the score ispartitioned intoS = L ∪R, whereL corresponds to thenote events of the left hand andR to the note events of theright hand. Starting with the model spectrogramY S
λ (Fig-ure 4a) we derive the model spectrogramsY L
λ andY Rλ using
Eqn. (5) (Figure 4b) and then the two masksML andMR
using Eqn. (6) (Figure 4c). Applying the two masks to theoriginal audio spectrogramY , we obtain the estimated mag-nitude spectrogramsY L and Y R (Figure 4d). Finally, ap-plying the Griffin-Lim based spectrogram inversion yieldsthe reconstructed audio signalsxL andxR (Figure 4e).
4. EXPERIMENTS
In this section, we report on systematically conducted ex-periments to illustrate the potential of our method. To thisend, we created a database consisting of seven representa-tive pieces from the Western classical music repertoire, seeTable 1. Using only freely available audio and score data al-
248
12th International Society for Music Information Retrieval Conference (ISMIR 2011)
Table 1. Pieces and audio recordings (with identifier) used in ourexperiments.
lows for a straightforward replication of our experiments.Here, we used uninterpreted score-like MIDI files fromthe Mutopia Project1 (MUT), high-quality audio recordingsfrom the Saarland Music Database2 (SMD) as well as dig-itized versions of historical gramophone and vinyl record-ings from the European Archive3 (EA).
In a first step, we indicate the quality of our approachquantitatively using synthetic audio data. To this end, weused the Mutopia MIDI files to create two additional MIDIfiles for each piece using only the notes of the left and theright hand, respectively. Using a wave table synthesizer,we then generated audio recordings from these MIDI fileswhich are used as ground truth separation results in the fol-lowing experiment. We denote the corresponding magni-tude spectrograms byY L and Y R, respectively. For ourevaluation we use a quality measure based on the signal-to-noise ratio (SNR)4 . More precisely, to compare a referencemagnitude spectrogramYR ∈ R
K×N≥0 to an approximation
YA ∈ RK×N≥0 we define
SNR(YR, YA) := 10 · log10
∑k,n YR(k, n)2
∑k,n (YR(k, n)− YA(k, n))2
.
The second and third column of Table 2 show SNR val-ues for all pieces, where the ground truth is compared tothe estimated spectrogram for the left and the right hand.For example, the left hand SNR for ‘Chop28-15’ is17.79whereas the right hand SNR is13.35. The reason the SNRbeing higher for the left hand than for the right hand is thatthe left hand is already dominating the mixture in terms ofoverall loudness. Therefore, the left hand segregation is perse easier compared the the right hand segregation. To indi-cate which hand is dominating in a recording, we addition-ally give SNR values comparing the ground truth magnitudespectrogramsY L andY R to the mixture magnitude spectro-gramY , see column six and seven of Table 2. For examplefor ‘Chop28-15’,SNR(Y L, Y )=3.48 is much higher com-pared toSNR(Y R, Y )=−2.47 thus revealing the left handdominance.
1 http://www.mutopiaproject.org2 http://www.mpi-inf.mpg.de/resources/SMD/3 http://www.europarchive.org4 Even though SNR values are often not perceptually meaningful, they
at least give some tendencies on the quality of separation results.
Table 2. Experimental results using ground truth data consistingof synthesized versions of the pieces in our database.
Using synthetic data, the audio recordings are alreadyperfectly aligned to the MIDI files. To further evaluate theinfluence of the music synchronization step, we randomlydistorted the MIDI files by splitting them into20 segmentsof equal length and by stretching or compressing each seg-ment by a random factor within an allowed distortion range(in our experiments we used a range of±50%). The resultsfor these distorted MIDI files are given in column four andfive of Table 2. Here, the left hand SNR for ‘Chop28-15’decreases only moderately from17.79 (prealigned MIDI)to 17.56 (distorted MIDI), and from13.35 to 13.18 for theright hand. Similarly, the average SNR also decreases mod-erately from13.45 to 13.27 for the left hand and from12.05to11.90 for the right hand, which indicates that our synchro-nization works robustly in these cases. The situation in realworld scenarios becomes more difficult, since here the noteevents of the given MIDI may not correspond one-to-one tothe played note events of a specific recording. An examplewill be discussed in the next paragraph, see also Figure 5.
As mentioned before, signal-to-noise ratios and similarmeasures cannot capture the perceptual separation quality.Therefore, to give a realistic and perceptually meaningfulimpression of the separation quality, we additionally pro-vide a website5 with audible separation results as well asvisualizations illustrating the intermediate steps in ourpro-cedure. Here, we only used real, non-synthetic audio record-ings from the SMD and EA databases to illustrate the per-formance of our approach in real world scenarios. Lis-tening to these examples does not only allow to quicklyget an intuition of the method’s properties but also to effi-ciently locate and analyze local artifacts and separation er-rors. For example, Figure 5 illustrates the separation pro-cess for ‘BeetOp111’ using an interpretation by Egon Petri(European Archive). As a historical recording, the spectro-gram of this recording (Figure 5c) is rather noisy and revealssome artifacts typical for vinyl recordings such as rumblingand cranking glitches. Despite these artifacts, our modelapproximates the audio spectrogram well (w.r.t. to the eu-clidean norm) in most areas (Figure 5d). Also the resulting
Figure 5. Illustration of the separation process for ‘BeetOp111’.(a): Score corresponding to the first two measures.(b): MIDIrepresentation (Mutopia Project).(c): Spectrogram of an interpre-tation by Petri (European Archive).(d): Model spectrogram afterparameter estimation.(e): Separation maskML. (f): Estimatedmagnitude spectrogramY L. The area corresponding to the funda-mental frequency of the trills in measure one is indicated using agreen rectangle.
separation results are plausible, with one local exception.Listening to the separation results reveals that the trillsto-wards the end of the first measure were assigned to the leftinstead of the right hand. Investigating the underlying rea-sons shows that the trills are not correctly reflected by thegiven MIDI file (Figure 5b). As a consequence, our score-informed approach cannot model this spectrogram area cor-rectly as can be observed in the marked areas in Figures 5cand 5d. Applying the resulting separation mask (Figure 5e)to the original spectrogram leads to the trills being misas-signed to the left hand in the estimated magnitude spectro-gram as shown in Figure 5f.
5. CONCLUSIONS
In this paper, we presented a novel method for the decompo-sition of a monaural audio recording into musically mean-
ingful voices. Here, our goal was to extend the idea of aninstrument equalizer to a voice equalizer which does notrely on statistical properties of the sound sources and whichis able to emphasize or attenuate even single notes playedby the same instrument. Instead of relying on prealignedMIDI files, our score-informed approach directly addressesalignment issues using high-resolution music synchroniza-tion techniques thus allowing for an adoption in real worldscenarios. Initial experiments showed good results usingsynthetic as well as real audio recordings. In the future,we plan to extend our approach with an onset model whileavoiding the drawbacks discussed in [9].
Acknowledgement. This work has been supported by theGerman Research Foundation (DFG CL 64/6-1) and theCluster of Excellence on Multimodal Computing and Inter-action at Saarland University.
6. REFERENCES
[1] J.-L. Durrieu, G. Richard, B. David, and C. Fevotte. Source/filter modelfor unsupervised main melody extraction from polyphonic audio sig-nals.IEEE Transactions on Audio, Speech and Language Processing,18(3):564–575, 2010.
[2] S. Ewert and M. Muller. Estimating note intensities in music record-ings. InProceedings of the IEEE International Conference on Acous-tics, Speech, and Signal Processing (ICASSP), pages 385–388, Prague,Czech Republic, 2011.
[3] S. Ewert, M. Muller, and P. Grosche. High resolution audio synchro-nization using chroma onset features. InProceedings of IEEE In-ternational Conference on Acoustics, Speech, and Signal Processing(ICASSP), pages 1869–1872, Taipei, Taiwan, 2009.
[4] J. Ganseman, P. Scheunders, G. J. Mysore, and J. S. Abel. Source sepa-ration by score synthesis. InProceedings of the International ComputerMusic Conference (ICMC), pages 462–465, New York, USA, 2010.
[5] D. W. Griffin and J. S. Lim. Signal estimation from modified short-timeFourier transform.IEEE Transactions on Acoustics, Speech and SignalProcessing, 32(2):236–243, 1984.
[6] Y. Han and C. Raphael. Desoloing monaural audio using mixture mod-els. InProceedings of the International Society for Music InformationRetrieval Conference (ISMIR), pages 145–148, Vienna, Austria, 2007.
[7] T. Heittola, A. Klapuri, and T. Virtanen. Musical instrument recogni-tion in polyphonic audio using source-filter model for sound separa-tion. In Proceedings of the International Society for Music InformationRetrieval Conference (ISMIR), pages 327–332, Kobe, Japan, 2009.
[8] R. Hennequin, B. David, and R. Badeau. Score informed audio sourceseparation using a parametric model of non-negative spectrogram.In Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP), pages 45–48, Prague, CzechRepublic, 2011.
[9] K. Itoyama, M. Goto, K. Komatani, T. Ogata, and H. G. Okuno. In-strument equalizer for query-by-example retrieval: Improving soundsource separation based on integrated harmonic and inharmonic mod-els. InProceedings of the International Conference for Music Informa-tion Retrieval (ISMIR), pages 133–138, Philadelphia, USA, 2008.
[10] Y. Ueda, Y. Uchiyama, T. Nishimoto, N. Ono, and S. Sagayama.HMM-based approach for automatic chord detection using refined acous-tic features. InProceedings of the IEEE International Conference onAcoustics, Speech and Signal Processing (ICASSP), pages 5518–5521,Dallas, USA, 2010.
[11] J. Woodruff, B. Pardo, and R. B. Dannenberg. Remixing stereo mu-sic with score-informed source separation. InProceedings of the Inter-national Conference on Music Information Retrieval (ISMIR), pages314–319, 2006.
