Multimodal Presentation and Browsing of Music

David Damm∗, Christian Fremerey∗, Frank Kurth†, Meinard Müller‡, Michael Clausen∗

∗Department of Computer Science IIIUniversity of Bonn

53117 Bonn, Germany{damm,fremerey,clausen}@iai.uni-

bonn.de

†Research Establishmentfor Applied Science

53343 Wachtberg, Germanykurth@fgan.de

‡Max-Planck-Institutfür Informatik

66123 Saarbrücken, Germanymeinard@mpi-inf.mpg.de

ABSTRACTRecent digitization efforts have led to large music collections,which contain music documents of various modes compris-ing textual, visual and acoustic data. In this paper, wepresent a multimodal music player for presenting and brows-ing digitized music collections consisting of heterogeneousdocument types. In particular, we concentrate on musicdocuments of two widely used types for representing a mu-sical work, namely visual music representation (scanned im-ages of sheet music) and associated interpretations (audiorecordings). We introduce novel user interfaces for multi-modal (audio-visual) music presentation as well as intuitivenavigation and browsing. Our system offers high quality au-dio playback with time-synchronous display of the digitizedsheet music associated to a musical work. Furthermore, oursystem enables a user to seamlessly crossfade between vari-ous interpretations belonging to the currently selected mu-sical work.

Categories and Subject DescriptorsH.5.5 [Computer Applications]: Information Interfacesand Presentation—Sound and Music Computing

General TermsDesign

1. INTRODUCTIONRecent advances in Music Information Retrieval (MIR) havebrought up new possibilities for accessing, browsing and nav-igating in large collections of digital music. In this paper,we employ recently introduced MIR-techniques for musicsynchronization to develop novel user interfaces for multi-modal presentation, navigation and inter-document brows-ing of music documents belonging to a musical work basedon music synchronization.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICMI’08, October 20–22, 2008, Chania, Crete, Greece.Copyright 2008 ACM 978-1-60558-198-9/08/10 ...$5.00.

A musical work can be described by various formats anddocument types. Throughout the paper, we distinguish be-tween a musical work (in an abstract sense), its sheet mu-sic representation and a particular audio recording (a con-crete interpretation). In our scenario, for a musical work wehave a sheet music representation as well as associated au-dio recordings representing different interpretations by vari-ous musicians. While sheet music contains musical symbols,which describe the musical content visually and abstractsfrom a concrete realization, i.e. interpretation as well asinstrumentation, an audio recording retains a concrete real-ization of music, which describes the musical content acous-tically. Note that both sheet music and audio recordings, re-spectively, may be considered as two natural forms of musicrepresentation as they explicitly address the visual and audi-torial modalities respectively. For this reason, both of thoserepresentations are most widely employed by users for ac-cessing music, and corresponding multimodal user interfacesare of high importance. It turns out that the key challengein designing such interfaces and in suitably preprocessingthe underlying music documents is to find an appropriatecommon representation for both music modes coming fromdifferent domains in order to compare and relate the musicalcontent. Therefore the various document types have to bereducible to the same representation.

In the context of MIR, music synchronization denotes aprocedure which, for a given position in one representationof a musical work, determines the corresponding positionwithin another representation, e.g. the coordination of scoresymbols with audio data. Various synchronization taskshave been described in [1, 5, 6, 8]. In the following, we focuson the special scenario of SheetMusic-Audio synchronizationas well as Audio-Audio synchronization. SheetMusic-Audiosynchronization is used to align corresponding note eventswithin a sheet music representation of a musical work andtime regions within an associated interpretation. Audio-Audio synchronization is used to time-align correspondingnote events in various interpretations, i.e. different audiorecordings, of a musical work.

In this paper, we present novel user interfaces utilizingthese alignments to support a high-quality audio-visual pre-sentation of a musical work and an intuitive navigation bychoosing a particular measure within the sheet. Further-more the user has the option to seamlessly crossfade betweendifferent interpretations, which allows for a convenient com-parison of two or more interpretations. The Sheet MusicInterface (see Fig. 2) enables the user enjoying music in amultimodal way. On the one hand, it provides an automatic

score tracking while listening to a musical work. Further-more, the user has the option to navigate within the sheetmusic and choose a particular measure to change the play-back position. The Interpretation Switcher (see Fig. 4) al-lows the user to switch between several interpretations (au-dio recordings) of the current musical work. A user maylisten to one of the selected interpretations and then, at anytime during playback, switch to another interpretation. Theplayback in the target interpretation will start at the posi-tion that musically corresponds to the position inside thecurrent interpretation. In other words, a seamless transi-tion from one interpretation to another is performed whilemaintaining audio playback.

The rest of this paper is organized as follows. In Sect. 2,we give an outline of the involved MIR techniques used forSheetMusic-Audio synchronization as well as Audio-Audiosynchronization. Subsequently, in Sect. 3 and 4, we willintroduce our new, multimodal user interfaces. In particu-lar, in Sect. 3 we will present the user interface exploitingthe SheetMusic-Audio synchronization in order to facilitatemultimodal music presentation of a musical work as wellas an intuitive navigation. In Sect. 4 we will present theinter-document browser interface utilizing the Audio-Audiosynchronization to switch between various interpretations ofthe current musical work. Conclusions are given in Sect. 5.

2. UNDERLYING TECHNIQUESIn this section, we describe our approach to synchronize agiven sheet music representation of a musical work with oneor more associated audio recordings. The approach is basedon the background of techniques from MIR and audio signalprocessing. In particular, we use chroma-based features [2,6, 7] as mid-level representation for both, the sheet musicas well as the audio recordings, which is then used for thealignment of sheet music to audio as well as audio to audio.

The mid-level feature representation has to satisfy severalcritical requirements. On the one hand, it has to be ro-bust to semantic variations and transformation errors. Onthe other hand, it has to be characteristic enough to cap-ture distinctive musical aspects of the underlying musicalwork. It has been carried out that chroma-based featuresdo achieve these requirements. The chroma correspond tothe twelve traditional pitch classes of the equal-temperedscale [2]. In Western music notation, the chroma are com-monly indicated by the set {C, C#, . . . , B} consisting ofthe twelve pitch spelling attributes. Chroma-based featuresare well-known to reflect the phenomenon that human per-ception of pitch is periodic in the sense that two pitches areperceived as similar to the human auditory system if theydiffer by one or more octaves [2].

For an audio recording, the sample sequence is trans-formed into a sequence of normalized 12-dimensional chromavectors. Here, each vector embodies the local energy distri-bution among the twelve pitch classes. Based on signal pro-cessing techniques, a chroma representation can be obtainedusing either short-time Fourier analysis in combination withbinning strategies [2] or multirate filter banks [7]. Fig. 2 (c)shows an audio chromagram for the first few measures ofan audio recording (d) of the 3rd movement of Beethoven’spiano sonata no. 8, op. 13 (”Pathethique”). Chroma-basedfeatures absorb variations in parameters such as dynamics,timbre and articulation and describe the coarse harmonyprogression of the underlying audio signal. Note that this

Figure 1: Data types involved in SheetMusic-Audio synchronization for the first few measuresof Beethoven’s piano sonata no. 8, op. 13 (”Pa-thethique”), Rondo (3rd movement). (a) Sheet mu-sic. (b) Sheet music chromagram. (c) Audio chro-magram. (d) Audio recording. The SheetMusic-Audio linking structure (double-headed arrows) isobtained by aligning the two chromagrams.

is needed to compare an audio recording with a sheet musicrepresentation due to the absence of these parameters in thelatter.

The transformation of a sheet music representation intoa chromagram consists of several steps. In a first step, thescore data such as note events, i.e. onset times, pitchesand durations, the key and time signatures and other mu-sical symbols, are extracted using optical music recognition(OMR) [3, 4]. This process is similar to the well-known op-tical character recognition (OCR), where textual content isextracted from an image. Note that this process is error-prone and the recognition accuracy of the OMR processstrongly depends on the quality of the input image data aswell as the complexity of the underlying score. In the contextof this paper, we consider high quality scans of piano musicat a resolution of 600 dpi and 1 bit color depth (b/w). In ad-dition to the musical score data, the OMR process providesus with spatial information. In particular, we get the exactpixel coordinates of the extracted data as well as bar lineinformation. This allows us to localize all musical symbolswithin the sheet music. In a second step, a sequence of nor-malized 12-dimensional chroma vectors is synthesized fromthe OMR output, which consists of a sequence of the ex-tracted note events, encoded by parameters for pitch, onsettime and duration. Fig. 2 (b) shows a chromagram obtainedfrom a sheet music of the ”Pathethique”. The chromagramderived from a sheet music is computed by sliding across thetime axis with a temporal window while adding energy tothe chroma bands that correspond to pitches that are ac-tive during the current temporal window. A single temporalwindow equals a single chroma vector. For timing informa-

Figure 2: The Sheet Music Interface for multimodalmusic presentation and navigation. Synchronouslyto audio playback, corresponding musical measureswithin the sheet music are highlighted. A click on ameasure changes the playback position. Additionalcontrol elements on the bottom can be used to nav-igate through the currently selected musical work.

tion we assume a constant tempo of 100 bpm. Note thatthis particular choice of tempo is not crucial, because dif-ferences in tempo will be compensated in the subsequentsynchronization step.

Given two chroma sequences extracted from a sheet musicand an associated audio recording, respectively, one can usestandard algorithms based on dynamic time warping (DTW)to align the two sequences. There have been proposed sev-eral synchronization strategies, see, e.g. [1, 6, 8] and the ref-erences therein. Most of these approaches rely on some vari-ant of DTW. The main idea is to build up a cross-similaritymatrix by computing the pairwise distance between eachsheet music chroma vector and each audio chroma vector.Here, we use the inner product for the comparison. Anoptimum-cost alignment path is determined from this ma-trix via dynamic programming. The resulting path throughthe matrix encodes a spatial-temporal alignment of the sheetmusic and an associated audio recording. The spatial in-formation of the OMR output allows for a localization ofchroma vectors within the sheet music image, thus allowinga linkage between groups of note events and correspondingimage regions. Combining the spatial information with thesynchronization result of a sheet music chroma sequence andan audio chroma sequence, we have all linking informationneeded to track and highlight note events in a sheet musicwhile playing an associated audio recording.

The same approach used for SheetMusic-Audio synchro-nization is used to align two or more interpretations of amusical work, which may differ in considerable deviationsin tempo, note realization, dynamics and instrumentation.Here, the chroma sequences to align are obtained from thevarious audio recordings. For a more detailed description,see, e.g. [1, 6, 8].

3. SHEET MUSIC INTERFACEIn this section, we present our new Sheet Music Interfacefor presenting sheet music while playing back associated au-

Figure 3: The Thumbnail Browser allows the user toconveniently navigate through the currently selectedmusical work.

dio recordings, depicted in Fig. 2. Here, the main visu-alization mode is illustrated for two scanned pages of theabove example, Beethoven’s piano sonata no. 8, op. 13 (”Pa-thethique”). When starting audio playback, correspondingmeasures within the sheet music are synchronously high-lighted based on the linking information generated by theSheetMusic-Audio alignment described in Sect. 2. In Fig. 2,a region in the center of the right page, corresponding to theeight measure of the 3rd movement (Rondo), is currentlyhighlighted by a surrounding box. When reaching the endof an odd-numbered page during playback, pages are turnedover automatically. Additional control elements allow theuser to switch between measures of the currently selectedmusical work. The Sheet Music Interface allows to navigatethrough the sheets of music using piece- or page-numbersthat are located below the scanned pages. By clicking ona measure, the playback position is changed and the audiorecording is resumed at the appropriate time position. Anicon in the top left corner indicates which interpretation iscurrently used for audio playback. If more than one associ-ated audio recording is available for the currently active mu-sical work, the user may switch between those using an iconlist that is shown by clicking on the current icon. A moredetailed description of this feature is given in Sect. 4. Click-ing on the icon in the top right corner opens the ThumbnailBrowser. Using the Thumbnail Browser shown in Fig. 3, alocal context of pages around the current playback positionis displayed and may be used for navigation.

The technical background underlying the Sheet Music In-terface’s presentation and navigation capabilities is summa-rized as follows. The Sheet Music Interface utilizes synchro-nization data which links image regions within the sheetmusic and time regions within the selected audio record-ing. For highlighting regions in a sheet music while playingan associated audio recording, one has to choose a suitablespatial-temporal granularity. To be more robust against lo-cal OMR errors we choose a granularity where the displayedregion within the sheet music corresponds to one musicalmeasure. This leads to a stable and accurate synchroniza-tion of sheet music and an associated audio recording evenin the case of typical local OMR errors.

Figure 4: The Interpretation Switcher for a seamlesscrossfade from one interpretation to another. Syn-chronously to audio playback, the slider knobs ofthe various interpretations run along the time linebars. A click on a time line bar’s play symbol (left)changes the interpretation used for audio playback.

The Sheet Music Interface described above is a convenientway to enjoy a musical work in a multimodal way. On theone hand, the user can see the sheet music along with thecurrently played measure highlighted while listening to themusical work. On the other hand, the navigation within thesheet music gives an intuitive modality to search for specificparts and change the playback position respectively.

4. INTERPRETATION SWITCHERIn this section, we present our new Interpretation Switcherfor seamless transitions between different interpretations be-longing to the currently active musical work, depicted inFig. 4. In the Interpretation Switcher window, there arelisted various available interpretations belonging to the mu-sical work. Each of the interpretations is represented bya slider bar indicating the current playback position withrespect to the recording’s particular time scale. The inter-pretation that is currently used for audio playback, in thefollowing referred to as reference recording, is indicated bya red playback symbol located to the left of the slider bar.The slider knob of the reference recording moves at constantspeed while the slider knobs of the other recordings moveaccordingly to the relative tempo variations with respect tothe reference. The reference recording may be changed atany time simply by clicking on the respective playback sym-bol located to the left of each slider bar. The playback ofthe new reference recording then starts at the time positionthat musically corresponds to the last playback position ofthe former reference. This has the effect of seamlessly cross-fading from one interpretation to another while preservingthe current playback position in a musical sense. One canalso jump to any position within any of the recordings by di-rectly selecting a position of the respective slider. This willautomatically initiate a switch of reference to the respectiverecording.

The technical background underlying the InterpretationSwitcher’s switching capabilities is summarized as follows.The Interpretation Switcher exploits synchronization data

which temporally links the selected audio recordings. Thisdata is given as a table of time positions with each columnrepresenting one recording and each row representing mutu-ally corresponding time positions. In a preprocessing step,the synchronization data is generated automatically frompairs of audio recordings using the algorithm outlined inSect. 2.

The Interpretation Switcher described above assists theuser in detecting and analyzing the differences between sev-eral interpretations of a single musical work. In the exampledepicted in Fig. 4, the user may switch between three in-terpretations of piano sonata no. 8, op. 13 (”Pathethique”).Besides the musical differences between the individual inter-pretations, also differences regarding acoustics, loudness orequalization become apparent.

5. CONCLUSIONSIn this paper, we have introduced a multimodal way for ex-periencing music. Given the associated sheet music as wellas one or more associated interpretations of a musical work,the user has a very intuitive access to the musical contentbased on the synchronous presentation of two high qualityaudio-visual music representations. While one listens to aninterpretation of the musical work, the listener can visuallytrack the corresponding part within the sheet music repre-sentation. Furthermore the visual component offers an intu-itive way to search for specific parts within the musical work.For a musical work there may exist more than one interpre-tation. Our system offers the user to seamlessly crossfadefrom one interpretation to another while maintaining thecurrent playback position in a musical sense.

6. REFERENCES[1] V. Arifi, M. Clausen, F. Kurth, and M. Muller.

Synchronization of music data in score-, MIDI- andPCM-format. Computing in Musicology, 13, 2004.

[2] M. A. Bartsch and G. H. Wakefield. Audiothumbnailing of popular music using chroma-basedrepresentations. IEEE Trans. on Multimedia,7(1):96–104, 2005.

[3] D. Byrd and M. Schindele. Prospects for improvingOMR with multiple recognizers. In Proc. ISMIR, pages41–46, 2006.

[4] G. Choudhury, T. DiLauro, M. Droettboom,I. Fujinaga, B. Harrington, and K. MacMillan. Opticalmusic recognition system within a large-scaledigitization project.

[5] R. Dannenberg and C. Raphael. Music score alignmentand computer accompaniment. Special Issue, Commun.ACM, 49(8):39–43, 2006.

[6] N. Hu, R. Dannenberg, and G. Tzanetakis. Polyphonicaudio matching and alignment for music retrieval. InProc. IEEE WASPAA, 2003.

[7] M. Muller. Information Retrieval for Music andMotion. Springer, 2007.

[8] M. Muller, H. Mattes, and F. Kurth. An efficientmultiscale approach to audio synchronization. In Proc.ISMIR, pages 192–197, 2006.