Gesture Control of Music Systems - unifr.ch · PDF fileGesture Control of Music Systems...

Gesture Control of Music Systems

Frédéric BevilacquaIrcamReal Time Musical Interactions Team

[email protected]://imtr.ircam.fr

http://www.ircam.fr/?L=1






mailto:[email protected]

mailto:[email protected]



IRCAM- Real Time Musical Interactions

Plan

• Research Context

• Digital Musical Instruments

• Gesture and Music‣ Gesture Analysis/Recognition of Musicians Gestures

• Mapping between Gestures and Sounds‣ Gesture Following and Recognition

• Applications

Gesture CaptureSound Synthesis

analysis/synthesis

concatenative synthesis

physical model sensors video game interfaces

Digital Music Instruments

motion capture

in interaction paradigms out

soundanalysis

gestureanalysis

sound capture synchronization

gesture-sound mapping

sound synthesis

audio processing

visualization

Musical Digital Instruments


Contexts

Digital Musical Instruments

Sound SynthesisInterface

Interaction Design

Music Technology

Human Machine Interaction

“Gesture Research”, cognitive sciences

IRCAM- Real Time Musical Interactions7

Digital Music Instruments

• Instrument-like‣ replicate an acoustic instrument

• Instrument-inspired‣ gesture or interface inspired from an acoustic instrument, but the final

musical goal is different than the acoustic instrument

• Extended instrument, Augmented Instrument, Hyper Instrument‣ Acoustic instrument with additional sensors

• Alternate controller ‣ New design

Marcelo M. Wanderley and Philippe Depalle. 2004. "Gestural Control of Sound Synthesis". Proceedings of the IEEE, vol. 92, No. 4 (April), pp. 632-644

Eduardo R. Miranda and Marcelo M. Wanderley. New Digital Musical Instruments: Control and Interaction beyond the Keyboard, A-R Editions, Spring 2006


« Instrument-like »

clavier MIDI Keyboard

EWI Electronic Wind Controller (AKAI)

Marimba Lumina (Buchla)

http://fr.youtube.com/watch?v=FNIKY5kGwLg




« Instrument-inspired »

Violon MIDI - Suguru Goto


HyperCelloTod Machover / Yo-Yo Ma

(1991)

Augmented Instruments

Clarinette & DataGlove, Butch Rovan


Theremin, 1928


« Alternative controllers »

« The Hands », Michel Waisviz Le Méta-Instrument - Serge de Laubier

http://fr.youtube.com/watch?v=U1L-mVGqug4




« The Hands », Michel Waisviz

Georgia Tech’s Guthman Musical Instrument Competition (2009)

Jaime Oliver's Silent Drum

the Slabs, David Wessel (CNMAT, Berkley)

Georgia Tech’s Guthman Musical Instrument Competition (2009)


?

Commercial interfaces


Stanford Laptop Orchestra (SLOrk)

http://mopho.stanford.edu/

http://mopho.stanford.edu

http://mopho.stanford.edu

IRCAM - Real Time Musical Interactions

Da Fact


reactable

http://www.reactable.com/

http://www.reactable.com

http://www.reactable.com


reactable


Installation Grainstick

• Cité des Sciences Paris

Pierre Jodlowski Raphaël Thibault

Ircam


Applications

• Music & New Media‣ professional level, music performance, composition‣ music pedagogy‣ music game

• HCI: interaction paradigms using “expressive gestures”

• Rehabilitation (?)Sonification of gesture/action (?)


Links to the HCI field

• Notion of embodied interaction ‣ P. Dourish Where The Action Is: The Foundations of

Embodied Interaction, MIT Press‣ M. Leman Embodied Music Cognition and Mediation

Technology, MIT Press

• Tangible interfaces, augmented reality

• Affective computing

• Collaborative and distributed interaction


Bill Buxton

• http://www.billbuxton.com/buxtonIRGVideos.html

• http://www.youtube.com/watch?v=Arrus9CxUiA

http://www.billbuxton.com/buxtonIRGVideos.html




http://www.youtube.com/watch?v=Arrus9CxUiA

http://www.youtube.com/watch?v=Arrus9CxUiA


Musical Interfaces

• action-perception loop

• importance of timing and synchronization‣ requirements: low latency (< 10 ms)

• from triggering events...to using continuous gestures

• notion of expressivity: measure of “how” is a gesture performed

• notions of “goal” and “efficiency” different than in standard HCI

32 CHAPTER 3. ACTION – SOUND

extensively in electroacoustic music. Here new technology has opened for creating allsorts of new "instruments" that challenge our ecological knowledge. One example is thephysically informed (and inspired) sonic model blotar, which is a combination of a flute,an electric guitar, and a mandolin (Trueman et al., 2001). Hearing such sounds may evokeseveral different and opposing mental images, and thereby open for interesting musicalexperiences. Yet another example of the creative use of action-sound relationships inmusic is the commonly used reversed cymbal sounds in electronic dance music. Suchreversed sounds play with our ecological knowledge of prefix, excitation and suffix, andcan lead to an upbeat and alert feeling which works well on the dance floor.

All in all, practical and creative action-sound design may be seen as two opposingdesign strategies, as shown in Figure 3.8. While the practical side is mainly focusing onease of use, the creative side is focusing on constructing new and interesting relation-ships. From a perceiver’s point of view, I find that much of the practical action-sounddesign found in electronic devices is often boring and uninspiring, while creative de-signs are often too confusing. A challenge here is to balance between the two axes, sothat practical action-sound designs may also feel interesting, and creative designs not toobewildering.

Practical

Creative

Easy to

use Interesting

to use

Figure 3.8: I see practical and creative action-sound designas opposing design strategies. Practical designs are often easyto use but may be boring, while creative designs are often moreinteresting yet can be bewildering.

3.4 SummaryIdeas from embodied music cognition presented in the previous chapter formed the basisfor the discussion of action-sound couplings and relationships elaborated in this chap-ter. I argue that our perception of sound-producing actions and the resultant sounds arebased on both the action and the sound, and that our mental imagery is based on theaction, the sound and the coupling between them. Furthermore, I suggest that our abil-ity to make predictions about the outcome of sound-producing actions arise from ourecological knowledge of action-sound couplings, and that these expectations are basedon the action-sound palette afforded by the object-action-object system. This enablesus to predict the sonic result of a sound-producing action we only see, or imagine thesound-producing action of a sound we only hear.

Similarly, I believe that ecological knowledge about action-sound couplings alsoguide our perception of artificially created action-sound relationships. Thus, if we wantto design more practical action-sound relationships, they should be based on qualities

from A.R.Jensenius PhD , 2007

27

"Clearly, electronic music systems allow much freedom for the performer, because the mappings between control units, on the one hand, and some production units, on the other hand, are not constrained be any biomechanical regularities. (...). However, as most electronic music performers know, it is exactly this freedom of mapping that may disturb the sens of contact and of non-mediation".

"Can we find a way of interacting with machines so that artistic expression can be fully integrated with contemporary technologies? »

Marc Leman, Embodied Music Cognition and Mediation Technology, MIT Press.

Gesture and Music


Gesture and Music

Some references:

• Cadoz, C. and M. M. Wanderley, Gesture - Music, in Trends in Gestural Control of Music, M. M. Wanderley and M. Battier, Editors. 2000, Ircam - Centre Pompidou: Paris, France. p. 71--94.

• Jensenius, A. R., M. M. Wanderley, R. I. Godoy and M. Leman (2010). Concepts and Methods in Research on Music-related Gestures. In Godøy, R. I. and M. Leman (Eds.), Musical Gestures: Sound, Movement and Meaning. Routledge.

http://musicalgestures.uio.no/



Types of Musical Gestures

4.3. MUSIC-RELATED MOVEMENT 45

Performance scene Home position Start position Performance

Figure 4.4: The performance scene is the imagined area in which performance can happen. Thehome position is the position where the musician is sitting (or standing) at ease before starting toperform. The start position is where the performance starts from, and the performance position isthe position(s) of the musician during performance.

and the perceiver, making it possible to identify where the sound-producing actions arecarried out.

Figure 4.5 also indicates the performance spaces of other other types of music-relatedmovements (as will be presented in the following sections). The idea of identifying thesespaces is to illustrate that we have a clear understanding of where different types ofmovements and actions should be carried out in relation to an object (e.g. an instrument).This knowledge of performance spaces for various types of music-related movementsalso helps us set up expectations when perceiving a performance. This is why we mayget surprised if a musician happen to perform outside of such conventional performancespaces, for example by playing with the fingers on the strings of the piano. Much mu-sical experimentation happen due to such exploration of the boundaries of establishedperformance spaces.

Ancillary,sound-accompanying,and communicative

Sound-producing

Sound-modifying

Figure 4.5: The action space can be seenas an imaginary box surrounding the spacein which movements can be carried out.Here the action spaces for various music-related movements are indicated, includingsound-producing and sound-modifying ac-tions, and ancillary, sound-accompanyingand communicative movements.

Jensenius, A. R., M. M. Wanderley, R. I. Godoy and M. Leman (2010). Concepts and Methods in Research on Music-

related Gestures. In Godøy, R. I. and M. Leman (Eds.), Musical Gestures: Sound, Movement and Meaning. Routledge.




Types of Musical Gestures

Jensenius, A. R., M. M. Wanderley, R. I. Godoy and M. Leman (2010). Concepts and Methods in Research on Music-

related Gestures. In Godøy, R. I. and M. Leman (Eds.), Musical Gestures: Sound, Movement and Meaning. Routledge.

48 CHAPTER 4. MUSIC-RELATED MOVEMENT

Excitation

Modification

Sound-producing Ancillary Communicative

Expressive

Theatrical

Entrained

Phrasing

Support

Figure 4.7: Examples of where different types of music-related movements (sound-producing, an-cillary and communicative) may be found in piano performance.

Sound-producing

Sound-facilitating

Sound-accompanying

Communicative

Sound-producing

Sound-facilitating

Sound-accompanying

Communicative

Figure 4.8: Dimension spaces illustrating how the music-related movements of a musician (left)and a dancer (right) may be seen as having different functions. Here the musician’s movementshave a high level of sound-producing and ancillary function, while the dancer’s movements have ahigh level of sound-accompanying and some communicative function.



Capturing Musician Motion

sensor attached on the bow

3D optical motion capture hybrid system

• E. Schoonderwaldt, N. Rasamimanana, F. Bevilacqua « Combining accelerometer and video camera:

Reconstruction of bow velocity profiles », NIME 2006

• F. Bevilacqua, N. Rasamimanana, E. Fléty, S. Lemouton, F. Baschet « The augmented violin project: research, composition and performance report » NIME 06

Violin bowing


Capturing Musician Gestures

• Direct capture of movement, pressure etc using sensors

• Indirect capture based on the sound analysis

+

+ + analysis

Martelé

Spiccato

Détaché

Bowing styles characterization

amax[a.u.]

2 violin players, 2 tempi (60 bpm, 120 bpm)dynamics (pp, mf, ff)

amin[a.u.]

amin

amax

amin

amax

amin

amax

N. Rasamimanana et al., GW 2005, Lecture Notes in Artificial Intelligence 3881, pp. 145–155, 2006.


Similar works

• PCA + KNN‣ D. Young. Classification of common violin bowing

techniques using gesture data from a playable measurement system. In in NIME 2008 Proceedings, 2009.


Bowing - Segmentation

time

Acc

eler

atio

n [a

.u]

détaché martelé

Δt Δt


Bowing recognition: Real time implementation (Max/MSP)

OSC in

1st order filtering

offset removal

peak detection

peak selection

amax, amin

knn recognition

gesture “intensity” computation

segmentation

OSC out

median filter

characterization/recognition


BogenLied -

mic accelerometers +wireless module

hub

soundcard

Receiver

spatialized sound(6 channels)

Sound processing

Gestureprocessing


Bowing styles

acceleration vs velocity

détaché martelé spiccato


Influence on bowing frequency

position

velocity

time

acceleration

acceleration

N. Rasamimanana et al., GW 2007, Lecture Notes in Artificial Intelligence

video


Bowing model

• Minimizing– Minimum impulse : trapezoïdal

“continuous control”

– Minimum jerk (discrete)“balistic control”

TrapJd

Original

mm

/sm

m/s

^2


Bowing style - scale

Détaché Martelé

Minimum impulse (Trapezoidal) Minimum Jerk

Finding the best model

N. Rasamimanana, F. Bevilacqua. « Effort-based analysis of bowing movements: evidence of anticipation effects ». Journal of New Music Research,


Gestural Co-articulation

detaché martelé

Minimum Jerk

Miminum Impulse


Co-articulation effect

• major difficulty for segmentation and characterization‣ using di-gesture ? (similarly to diphone)

• can be used to anticipate (towards intention ?)

• expressivity links to co-articulation

Gesture to Sound Mapping

Mapping

Wanderley, M. 2001. Performer-Instrument Interaction: Applications to Gestural Control of Music. PhD Thesis. Paris, France: University Pierre et Marie Curie - Paris VI

See also: •"Mapping Strategies in Interactive Computer Music." Organised Sound, 7(2), Marcelo Wanderley Ed.•Wanderley, M and Battier, M -editors. “Trends in Gestural Control of Music”. IRCAM, Centre Pompidou, 2000.


Mapping

• Low Level vs High Level

should be dealt with in these different modali-ties. A technology focusing on affect, emotion,expressiveness, and cross-modality interactionsis thus required.

Layered conceptual frameworkThe physical stimuli that make up an artistic

environment contain information about expres-siveness. That information can, to some extent,be extracted and then communicated among aMIEE’s virtual and real subjects. With multiplesensory modalities (auditory, visual, motoric/gestural), this allows the transmission of expres-siveness parameters from one domain to another—for example, from music (auditory) to computeranimation (visual), or from dance (motoric) tomusic (auditory).

Figure 1 shows a way of conceiving the trans-mission of cross-modal packages of expressive-ness. The signal-based level represents theanalysis and synthesis of physical properties (bot-tom). The symbolic level represents the descrip-tions of meanings, affects, emotions, andexpressiveness in terms of linguistic or visualsymbolic entities (top). The gesture-based levelrepresents spaces in which trajectories allow theconnection from signal-based descriptions tosymbolic-based descriptions (middle). In mostcases, the signal-based descriptions pertain to thesignal’s syntactical properties, while the symbol-based descriptions pertain to its semantic prop-erties. The latter may include cognitive, emotive,affective, and expressive evaluations.

The flow of packages of expressiveness may goin two directions (upward and downward).Taking the expressive hi-fi music system as anexample, physical properties of human move-ment (of the system’s user) may be extracted andgestures mapped as a trajectory on a space. Thattrajectory describes the expressive content interms of linguistic–semantic descriptors such ashow much the movement is fluent, smooth,heavy, rigid, and so on. Starting from this lin-guistic–semantic description (in the downwarddirection), a particular gesture-based trajectorymay be used to synthesize physical properties ofthat expressive content in another modality, inour case music (in terms of legato/staccato,amplitude, shapes of notes, and so on). The ges-ture-based mappings describe properties ofexpressiveness that are independent from anyparticular sensory modality. This level of map-ping introduces flexibility to the multimodal rep-resentational model.

Signal-based layerThe signal-based layer extracts the relevant

physical features for processing expressiveness.In the musical audio processing domain, Lesaffreet al.4 have worked out a useful taxonomy of con-cepts that gives a structured understanding ofthis layer (see Figure 2, next page).

Whether this taxonomy can be worked out interms of a set of predefined modules that extractthese features from musical audio signals isanother matter. Research5 in analysis of musicalexpressiveness (the upward direction in Figure 1)shows that a linear combination of predefinedacoustical features can only partly explain thelinguistic–semantical descriptions. We needmore research to figure out how we can extendthis Cartesian modeling approach to nonlinearmethods and imitative computational approach-es. Yet several of these predefined features areknown to work well in the context of musicalsynthesis6,7 (downward direction of Figure 1).

In the human movement and dance analysisdomain, we envisage a similar approach that dis-tinguishes among features calculated on differ-ent time scales. Also in this context, it makessense to differentiate among

! low-level features, calculated on a time inter-val of a few milliseconds (for example, one ora few frames coming from a video camera);

! mid-level features, calculated on a movementstroke, or motion phase, on time durations ofa few seconds; and

! high-level features that relate to the conveyedexpressive content (but also to cognitiveaspects) and refer to sequences of movementstrokes or motion (and pause) phases.

45

January–March 2005

Linguistic-based descriptions of semantical properties(meaning, affect, emotion, expressiveness, and so on)

Gesture-based descriptions as trajectories in spaces

Signal-based descriptions of the syntactical features

Figure 1. The layeredconceptual frameworkdistinguishes betweensyntax and semantics,and in between, aconnection layer thatconsists ofaffect/emotionexpressiveness spacesand mappings.

Antonio Camurri, Gualtiero Volpe, Giovanni De Poli, Marc Leman, "Communicating Expressiveness and Affect in Multimodal Interactive Systems," IEEE MultiMedia, vol. 12, no. 1, pp. 43-53, Jan. 2005


Mapping

• Spatial vs Temporal : ‣ « Spatial » : relationship independent of the temporal ordering of

data‣ « Temporel » : relationship between temporal processes

• Direct vs Indirect‣ Direct :

- sensor data directly connected to music parameters- relationship “manually” set

‣ Indirect- uses machine learning techniques to set the relationship


Mapping (Spatial)

• one-to-one

sensor data sound parameters


Mapping Musical Instruments

IDMIL lab, Mc Gill


Mapping

• one-to-many



Mapping

• many-to-one



Simple ou complexe mapping ?Hunt, A., and Kirk, R. 2000. "Mapping Strategies for Musical Performance." In M. Wanderley and M. Battier, eds. Trends in Gestural Control of Music. Ircam, Centre Pompidou.

The Physical Sliders Interface The Multiparametric Interface


Conclusions of Hunt and Kirk study

• The multiparametric interface allowed people to think gesturally, or to mentally rehearse sounds as shapes.

• The majority of users felt that the multiparametric interface had the most long-term potential.

• Several users reported that the multiparametric interface was “fun”.


Mapping

• Spatial vs Temporal : ‣ « Spatial » : relationship independent of the temporal ordering of

data‣ « Temporel » : relationship between temporal processes


Indirect Mapping using Machine Learning Techniques

• Neural Network‣ Mostly static postures

• Principal Component Analysis‣ Data dimension reduction

• Finite State Machine‣ Modeling sequences of postures

• DTW, HMM methods‣ Recognition of temporal profiles

Synchronization and recognition


Sydney Fels : Glove-TalkII

Fels, S. S. and Hinton, G. E. Glove-Talk: A neural network interface between a data-glove and a speech synthesizer. IEEE Trans. On Neural Networks, vol. 4, No. 1, 1993.

Artificial

Vocal Tract

(AVT)

Word

GeneratorGenerator

SyllableFinger

SpellingGenerator

Phoneme

Approximate time per gesture (msec)

10-30 100 130 200 600

SwitchesContact

Right Hand data

thumb and pinkie rotationwrist pitch and yaw

roll, pitch, yawevery 1/60 second

x, y, z

4 abduction angles

every 1/100 secondUser

10 flex angles

SpeechSynthesizer

Foot Pedal

Fixed PitchMapping

NetworkV/C Decision

VowelNetwork

ConsonantNetwork

Preprocessor

Fixed StopMapping

Combining

Function X

• adaptive Interface that Maps Hand Gestures to Speech• using neural network


Conducting gestures• Several works on conducting gestures‣ Study of professional conducting gesture‣ Beat detections, tempo, anticipation‣ Public Installation‣ Music Pedagogy

!"#$%& ' ()%*+,")- .") '/%0123+ 4"#/5!12#$ 6+715)+'#%89727

!"#$ %&&' ()*+ ,"-.//' 0&))#)* 1#&/' 2)3 42) 5+"$6&"78&3#2 9+:;.<#)* ,"+.;=>?0 @2$6&) A)#B&"7#<CDEFDG @2$6&)' ,&":2)C

{&"#$' *".&//' H#&/' I+"$6&"7}J$7K"L<6M22$6&)K3&

!"#$%!&$

!"#$%&$&% ' ()&*+(,$&% %"#,+-" '&'./#$# #/#,"0 1)- 2+3.$( #2'("#

2)#"# +&$4+" (5'.."&%"#6 7" 2-"#"&, !"#$%8 ' #)1,9'-" 1-'0":

9)-; ,5', "&'3."# '+,)0',$( -"()%&$,$)& '&* $&,"-2-",',$)& )1

()&*+(,$&% %"#,+-"#6 !"#$% $# '3." ,) -"()%&$<" 0+.,$2." ,/2"# )1

%"#,+-"# 9$,5 ='-/$&% ."=".# )1 *$1!(+.,/ 1)- ,5" +#"- ,) 2"-1)-08

1-)0 ' #,'&*'-* 1)+-:3"', 2',,"-&8 ,) #$02.$!"* +2:*)9& ()&*+(,:

$&% 0)="0"&,#8 ,) &) 2',,"-& ', '..6 !"#$% 2-)=$*"# '& ">,"&*'3."

.$3-'-/ )1 &'%()*' +'('!("*, .$&;"* ,)%",5"- $&,) ' *$-"(,"* '(/(.$(

%-'25? ,5"#" %-'25# -"2-"#"&, ,5" ='-$)+# ()&*+(,$&% 2',,"-&# '#

%"#,+-" -*"!.',6 @, -+&:,$0"8 !"#$% #"'-(5"# 1)- ,5" 3"#, 2-)!."

,) 0',(5 ' +#"-A# %"#,+-"# $& -"'.:,$0"8 '&* +#"# ' 3"', 2-"*$(:

,$)& '.%)-$,50 ,) 2-)=$*" -"#+.,# ', ,5" #+3:3"', ."=".8 $& '**$,$)&

,) )+,2+, ='.+"# #+(5 '# ,"02)8 %"#,+-" #$<"8 '&* ,5" %"#,+-"A#

%")0",-$( ("&,"-6 B&.$;" #)0" 2-"=$)+# '22-)'(5"#8 !"#$% *)"#

&), &""* ,) 3" ,-'$&"* 9$,5 #'02." *',' 3"1)-" +#"6 C+- 2-".$0:

$&'-/ +#"- ,"#,# #5)9 ,5', !"#$% 5'# ' 3"', -"()%&$,$)& -'," )1

)="- DEF6 !"#$% $# *"2.)/"* '# ,5" %"#,+-" -"()%&$,$)& #/#,"0

1)-/%',(*"08 '& $&,"-'(,$=" ()&*+(,$&% ">5$3$, ,5', )2"&"* $& ,5"

G",,/ G-$&& H5$.*-"&A# I+#"+0 $& I$.9'+;""8 BJ@ $& I'-(5

KEEL6

'()*+,-.

%"#,+-" -"()%&$,$)&8 ()&*+(,$&%8 #)1,9'-" %"#,+-" 1-'0"9)-;#

/0 12$%345&$132C-(5"#,-'. ()&*+(,$&% 5'# ' .)&% 5$#,)-/ $& 0+#$(8 9$,5 5$#:

,)-$('. #)+-("# %)$&% 3'(; '# 1'- '# ,5" 0$**." '%"#? $, 5'# '.#)

3"()0" '& )1,:">2.)-"* '-"' )1 ()02+,"- 0+#$( -"#"'-(56 H)&:

*+(,$&% $# 1'#($&',$&% '# '& $&,"-'(,$)& 0",'25)-8 3"('+#" )1 ,5"

5$%5 M3'&*9$*,5N )1 $&1)-0',$)& ,5', ")9# 3",9""& ,5" ()&*+(:

,)- '&* ,5" )-(5"#,-'6 @ ()&*+(,)-A# %"#,+-"# ()00+&$('," 3"',8

,"02)8 */&'0$(#8 ">2-"##$)&8 '&* "="& "&,-$"#O">$,# )1 #2"($!(

$&#,-+0"&, #"(,$)&#6 P+0"-)+# -"#"'-(5"-# 5'=" ">'0$&"* ()0:

2+,"- $&,"-2-",',$)& )1 ()&*+(,$&% %"#,+-"#8 '&* '22-)'(5"# -'&%:

$&% 1-)0 3'#$( ,5-"#5).* 0)&$,)-$&% )1 ' *$%$,'. 3',)&A# ="-,$:

('. 2)#$,$)&8 ,) 0)-" #)25$#,$(',"* '22-)'(5"# $&=).=$&% '-,$!($'.

&"+-'. &",9)-;# '&* Q$**"& I'-;)= I)*".#8 '&* "="& '&'./<:

$&% *',' 1-)0 0+.,$2." #"&#)-# 2.'("* )& ,5" ,)-#)8 5'=" 3""&

2-)2)#"*6

R"-0$##$)& ,) 0';" *$%$,'. )- 5'-* ()2$"# )1 '.. )- 2'-, )1 ,5$# 9)-; 1)-2"-#)&'. )- (.'##-))0 +#" $# %-'&,"* 9$,5)+, 1"" 2-)=$*"* ,5', ()2$"# '-"&), 0'*" )- *$#,-$3+,"* 1)- 2-)!, )- ()00"-($'. '*='&,'%" '&* ,5', ()2$"#3"'- ,5$# &),$(" '&* ,5" 1+.. ($,',$)& )& ,5" !-#, 2'%"6 S) ()2/ ),5"-9$#"8)- -"2+3.$#58 ,) 2)#, )& #"-="-# )- ,) -"*$#,-$3+," ,) .$#,#8 -"4+$-"# 2-$)-#2"($!( 2"-0$##$)& '&*O)- ' 1""612/3 456 T+&" U:V8 KEEL8 R'-$#8 W-'&("H)2/-$%5, -"0'$&# 9$,5 ,5" '+,5)-X#Y6

6789,( /: !"#$%&'(; <= 7=>(,<?>7@( ?+=-9?>7=8 (AB7C7> D+,

?B7E-,(= >B<> *( -(@(E+F(-; <> >B( "(>>) ",7== &B7E-,(=G.

H9.(9I 7= H7E*<9J((; 5#!0 )*'%' "++#"&$ ,'-&%#$. '/ %*#

0#%%. 0&122 3*145&#26$ !-$#-7 12 !148"-9##: ;<: =>?@

C+- 9)-; $# 0),$=',"* 3/ -"#"'-(5 )& &)=". ()02+,"- 0+#$(

'&* 0+.,$0"*$' $&,"-1'("# 1)- 2+3.$( #2'("# #+(5 '# 0+#"+0#

X#"" W$%+-" ZY8 '&* !"#$% 3+$.*# )& )+- 2-$)- ">2"-$"&(" 9$,5 *":

#$%&$&% $&,"-'(,$=" )-(5"#,-'. ()&*+(,$&% ">5$3$,#8 $&(.+*$&%7'*8

,"#%. 9*!:',(*%8 '& ">5$3$, 1)- ,5" QCBJ[ CW IBJ\H $& ]$"&&'

X())-*$&',"* 3/ I'> I+̂5.5'̂+#"-8 &)9 ', !'-0#,'*, B&$="-#$,/Y

_Z` '&* ' ()..'3)-',$)& 9$,5 S"-"#' I'--$& P';-' )& ;")<*' (:'

="#+)!("*8 ' (5$.*-"&A# ">5$3$, 1)- ,5" G)#,)& H5$.*-"&A# I+:

#"+0 _D`6 C+- #/#,"0# '..)9 ,5" +#"- ()&,-). )="- ,"02)8 3/

0';$&% 1'#,"- )- #.)9"- %"#,+-"#? =).+0"8 3/ 0';$&% .'-%"- )-

#0'.."- %"#,+-"#? '&* $&#,-+0"&, "025'#$#8 3/ *$-"(,$&% ,5" %"#:

,+-"# ,)9'-*# #2"($!( '-"'# )1 ' =$*") )1 ,5" )-(5"#,-' )& ' .'-%"

*$#2.'/ X$&#,-+0"&, "025'#$# $# &), #+22)-,"* $& ;")<*' (:' ="#8

+)!("*Y6 !"#$%&$&% ' %"#,+-" -"()%&$,$)& #/#,"0 1)- ' 0+#"+0:

,/2" "&=$-)&0"&, 2)#"# +&$4+" '&* $&,"-"#,$&% (5'.."&%"#8 2-$:

0'-$./ 3"('+#" 0+#"+0 =$#$,)-# 5'=" ' 9$*" -'&%" )1 ">2"-$"&("

9$,5 ()&*+(,$&%6 I)-")="-8 ,5"-" $# .$,,." ,) &) )22)-,+&$,/ ,)

"$,5"- ,-'$& ' +#"- ,) +#" ,5" #/#,"08 )- ,) ,-'$& ,5" #/#,"0 ,) '

#2"($!( +#"-? ' 0+#"+0 )& ' 3+#/ *'/ 0'/ #"" )="- ZEEE =$#$:

,)-#8 '&* #) ' =$#$,)- 9$.. #2"&*8 )& '="-'%"8 ."## ,5'& )&" 0$&+,"

', '& ">5$3$,6

\& ,5$# 2'2"-8 9" 2-"#"&, !"#$%8 ' #/#,"0 1)- ?+=*+(,$&%

8"#,+-" <&'./#$#6 B&.$;" (+--"&, #/#,"0#8 !"#$% *)"# &), -"#,-$(,

,5" +#"- ,) ()&*+(, $& ' #2"($!( 9'/8 &)- *)"# ,5" #/#,"0 $,#".1

-"4+$-" ,-'$&$&% ,) ,+&" $,#".1 ,) ' +#"-A# #2"($!( 0)="0"&,#? $&:

#,"'*8 $, ()&,$&+)+#./ "='.+',"# +#"- %"#,+-"# '%'$&#, ' #", )1 %"#:

,+-" -*"!.',8 95$(5 '-" "&()*"* 9$,5 ,5" (5'-'(,"-$#,$( 1"',+-"#

)1 2'-,$(+.'- ,/2"# )1 %"#,+-"#8 '&* +#"# ,5" 3"#,:0',(5$&% 2-)!."

,) ">,-'(, $&1)-0',$)& #+(5 '# 3"',8 ,"02)8 #$<"8 '&* ("&,"- 1-)0

!"#$%%&'()*+#,+-.%+/001+2(-%"(3-'#(34+5#(,%"%($%+#(+6%7+2(-%",3$%*+,#"+89*'$34+:;<"%**'#(+=628:01>?+!3"'*?+@"3($%

/10

E. Lee, I. Grüll, H. Kiel, and J. Borchers. conga: a framework for adaptive conducting gesture analysis. In NIME ’06: Proceedings of the 2006 conference on New interfaces for musical expression, pages 260–265, Paris, France,

!2

x

y

"

" beat

Rotate

!

BounceDetector

tempoRotateBounce

Detector

State[S1]

0.00

State[S2]

0.12

State[S3]

0.31

State[S4]

0.63

State[S5]

0.84

State Machine

1

32

4

S1

S2

S3

S5

S4

ZeroCrossing and

=+

ZeroCrossing and

=+

ZeroCrossing and

=+

S1

0.00

S4

0.63

S3

0.31

S2

0.12S5

0.84

!"#$%& '( )*& +&,- !#$%& .*/0. -*& !"#$% #%12* ,/% -*& !/$%34&1- 5&$-%1+36&#1-/ #&.-$%& 2%/!+&7 !"8& ,&1-$%&. 1%& 9&-&:-&9;

0*":* 1%& $.&9 -/ -%"##&% -*& 2%/#%&.. /, 1 .-1-& <1:*"=& -*1- 1+./ 1:-. 1. 1 >&1- 2%&9":-/%7 )*& "=2$- -/ -*& .-1-& <1:*"=& ". -*&

:$%%&=- 2%/#%&.. ?@ -/ AB /, -*& >1-/= 1. "- </8&. -*%/$#* /=& :/<2+&-& :C:+& /, -*& #&.-$%&; .-1%-"=# 1- -*&!%.- >&1-7 )*& %"#*-

!#$%& .*/0. -*& :/%%&.2/=9"=# >&1- 21--&%= -*1- ". -%1:D&9E =$<>&%&9 :"%:+&. "=9":1-& >&1-.; .F$1%&9 +1>&+. "=9":1-& -*& ,&1-$%&.

-*1- 1%& -%1:D&9 1=9 -*& .-1-& -*1- -*&C :/%%&.2/=9 -/7

)1>+& A( G$<<1%C /, +1-&=:C %&.$+-. ,/% -*& H23I/0= 2%/!+&7

!"#$ %&' (#)*+ ,-*). /01#234 ,)".

562 507 %&'

8 9: ;< 88= ><

? 9@ <; 88= @<

; >< @8 8:: 88;

: 88: @8 8:: 88@

9 8;A 8A@ 8;9 8??

*$#"#21 "+)# 3B0CC#2'#" DE$62' 6)*C#)#21016+2F G+$ #70)*C#H

D010 I$+) 0 J03+) 10-C#1 6" $#C016&#C4 B6'B $#"+CE16+2 02D 2+6"#K

I$##H 3+)*0$#D 1+ 1B# LE3BC0 /6'B1262' MMH NB63B B0" 0 $#"+CE16+2

+I +2C4 8?@ 62 -+1B N6D1B 02D B#6'B1F O010 I$+) 1B# LE3BC0 302

0C"+ -# PE61# 2+6"4H 02D N# #7*#$6)#21#D N61B D6II#$#21 14*#" +I

!C1#$62' 1+ 3+)*#2"01#F L0"#D +2 1B#"# #7*#$6)#21"H N# I+E2D

0 3+)-62016+2 +I B4"1#$#"6" !C1#$62' 02D 0 ;? *+621 Q02262' !CK

1#$ 1+ D#2+6"# 1B# D010 '6&#" 1B# -#"1 $#"EC1"F !2I+$1E201#C4H 1B6"

Q02262' !C1#$ 0C"+ 0DD" -#1N##2 : 1+ 8A "0)*C#" +I C01#234 1+

1B# +&#$0CC "4"1#) R;< 1+ >A )"SH 02D N# 0$# C++T62' 621+ 0C1#$20K

16&# )#1B+D" 1+ $#DE3# 1B6" C01#234 N61B+E1 3+)*$+)6"62' +&#$0CC

033E$034F

'7 JKL6HL)MN5J# 3+2DE31#D "+)# *$#C6)620$4 1#"162' N61B E"#$" 1+ #&0CE01#

!"#$%U" 033E$034 02D $#"*+2"#F J# 0"T#D !&# E"#$" RI+E$ )0C#H

+2# I#)0C#S 1+ 3+2DE31 E"62' E*KD+N2 )+&#)#21"H 02D 1B$##

E"#$" R0CC )0C#S 1+ 3+2DE31 E"62' 1B# I+E$K-#01 2#E1$0CKC#'01+ *01K

1#$2F (B# E"#$" 3+2DE31#D I+$ 0**$+76)01#C4 ;A "#3+2D" #03BF

(B# 1B$## E"#$" 3+2DE3162' 1B# I+E$K-#01 *011#$2 N#$# 0C$#0D4

"+)#NB01 *$+!36#21 N61B 1B# '#"1E$# *$6+$ 1+ 1B# #7*#$6)#21F

(B# "4"1#) "10$1" -4 D#I0EC1 E"62' 1B# J6''C# '#"1E$# *$+!C#V

I+$ 0CC !&# E"#$"H 1B# "4"1#) "N613B#D 1+ 1B# !*KO+N2 *$+!C#

N61B62 1B# !$"1 1N+ -#01"F %I1#$ 1B01H !"#$% D6D 2+1 I0C"#C4 D#1#31

024 -#01"H 2+$ )6"" 024 -#01"H 62 1B# E"#$U" '#"1E$#" R8AAW $#3+'K

2616+2 $01#SF J# 0C"+ )#0"E$#D !"#$%U" +&#$0CC C01#234 -4 )#0K

"E$62' 1B# 16)# D6II#$#23# -#1N##2 NB#2 1B# E"#$ )0$T" 0 -#01H

02D NB#2 61 6" D#1#31#D -4 !"#$%V 1B# $#"EC1" 0$# "E))0$6X#D 62

(0-C# 8F

Y623# 1B# ")++1B62' 1B01 N# 0**C4 1+ 1B# LE3BC0 -01+2 D010 62K

1$+DE3#" 02 0&#$0'# <; )" D#C04H N# #"16)01# !"#$%U" C01#234 1+

-# -#1N##2 ?; 02D 9> )"F (B#$# 0C"+ 0**#0$" 1+ -# 0 3+$$#C016+2

)1>+& O( G$<<1%C /, +1-&=:C %&.$+-. ,/% -*& !/$%34&1-

5&$-%1+36&#1-/ 2%/!+&7

!"#$ %&' (#)*+ ,-*). /01#234 ,)".

562 507 %&'

8 8A: ?= <=9 8=9

? >< =? <<< ?A;

; >@ 8@ ??9 8A=

-#1N##2 1#)*+ 02D C01#234H 0C1B+E'B )+$# D010 *+621" N+ECD -#

$#PE6$#D 1+ )0T# 0 3+23CE"6&# "101#)#21F

G+$ 1B# G+E$KL#01 Z#E1$0CK/#'01+ *011#$2H N# I+E2D 1B01 I+$

+2# E"#$H !"#$% I#CC -03T 1+ 1B# !*KO+N2 *$+!C# @W +I 1B# 16)#H

02D I06C#D 1+ D#1#31 B6" -#01" <W +I 1B# 16)#F G+$ 1B# +1B#$ 1N+

E"#$"H !"#$% "104#D 62 1B# G+E$KL#01 *$+!C# 8AAW +I 1B# 16)#H

02D D6D 2+1 I06C 1+ D#1#31 024 +I 1B#6$ -#01"F (B# C01#234 $#"EC1"

0$# "E))0$6X#D 62 (0-C# ?F

(B# )076)E) C01#236#" I+$ E"#$" 8 02D ? N#$# *0$163EC0$C4

B6'BV 0 3C+"#$ 020C4"6" +I 1B# D010 "B+N#D 1B01 1B#"# B6'B C01#2K

36#" +33E$$#D 3+2"6"1#21C4 +2 -#01 ;H 02D "+)#16)#" I+$ -#01 :F

[2# *+""6-C# #7*C02016+2 6" 1B01 1B# E"#$"U E2I0)6C60$614 N61B 1B#

I+E$K-#01 '#"1E$# 3+2IE"#D 1B# -#01 *$#D631+$H $#"EC162' 62 !"#$%

-#B0&62' E2*$#D6310-C4F G+$ E"#$ ;H NB+ 6" 1B# )+"1 I0)6C60$ N61B

1B# '#"1E$#H !"#$% I0$#D "6'26!3021C4 -#11#$F %'062H N# -#C6#&#

1B01 )+$# D010 *+621" 02D D#106C#D 020C4"6" N+ECD -# $#PE6$#D 1+

)0T# 0 3+23CE"6&# "101#)#21F

P7 !H)HQJ RNQSJ# B0&# 6D#216!#D 0 2E)-#$ +I 0$#0" 1B01 N# 0$# 0316&#C4 *E$K

"E62' 1+ IE$1B#$ 1B# D#&#C+*)#21 +I !"#$%\

T/%& #&.-$%& 2%/!+&.( J# B0&# 3E$$#21C4 6)*C#)#21#D 1B$##

'#"1E$# *$+!C#" 62 !"#$%H NB63B 0C$#0D4 6CCE"1$01# 1B# 30*0-6C616#"

02D *+1#2160C I+$ 1B# I$0)#N+$TF Q+N#&#$H +2C4 +2# +I 1B#"# 6"

031E0CC4 0 $#0C 3+2DE3162' '#"1E$#H 02D 1BE" N# N+ECD C6T# 1+ 62K

3+$*+$01# )+$# *$+I#""6+20C 3+2DE3162' "14C#"H "E3B 0" 1B# I+E$K

-#01 #7*$#""6&#KC#'01+ *011#$2 "B+N2 62 G6'E$# ?F

M<2%/8&9 2%/!+& .&+&:-/%( %" N# 623+$*+$01# )+$# '#"1E$#

*$+!C#"H N# N6CC 201E$0CC4 B0&# 1+ 6)*$+&# 1B# *$+!C# "#C#31+$ 0"

N#CCF G+$ #70)*C#H 1B# I+E$K-#01 2#E1$0CKC#'01+ 02D 1B# I+E$K-#01

IECCK"103301+ B0&# &#$4 "6)6C0$ "B0*#"H -E1 1B#6$ *C03#)#21 +I 1B#

-#01 6" D6II#$#21H 0" 6" 1B# N04 62 NB63B 1B#4 0$# #7#3E1#DF

6/0&% +1-&=:C(JB6C# 1B# 3E$$#21 C01#234 621$+DE3#D -4 !"#$%

!"#$%%&'()*+#,+-.%+/001+2(-%"(3-'#(34+5#(,%"%($%+#(+6%7+2(-%",3$%*+,#"+89*'$34+:;<"%**'#(+=628:01>?+!3"'*?+@"3($%

/1A


“Multimodal Music Stand”

Overholt, D., Thompson, J., Putnam, L., Bell, B., Kleban, J., Sturm, B., and Kuchera-Morin, J. 2009. A multimodal system for gesture recognition in interactive music performance. Comput. Music J. 33, 4 (Dec. 2009), 69-82

Figure 3. Prototype MMSSwith electric-field-sensorantennas mounted at thecorners of the stand.

Detection layer, where higher-level gestures arederived.

Electric-Field Sensing

The Multimodal Music Stand system incorporatesfour electric-field sensors (Mathews 1989; Boulangerand Mathews 1997; Paradiso and Gershenfeld 1997),as shown in Figure 3. These are used as part of themultimodal gesture-detection system and also asinput sources for the control of continuously variablemusical parameters, such as sound brightness ordensity. The electric-field sensors capture bodily andinstrumental gestures (they are sensitive to both)made by the performer, which are tracked via thefour sensor antennas.

The electric-field-sensing technique is based onthe original Theremin circuit topology (Smirnov2000), but all timing calculations are done en-tirely in the digital domain. Whereas Theremin’scircuit designs utilized analog heterodyning tech-niques, the MMSS only uses the “front end” ofthis type of analog circuit. The remaining logic isaccomplished through the measurement of high-frequency pulse widths using custom-designedfirmware on the CREATE USB Interface (Overholt2006).

The data from each of the independent electric-field sensors is received in Max/MSP/Jitter, aspictured in Figure 4, mean-filtered, and sent on to

the multimodal detection layer via OSC. Using fourchannels of sensing makes it possible for the MMSSto provide full three-dimensional input. In contrastwith the Theremin’s dual-channel approach, three-dimensional proximity can be sensed. The overallintensity of all four antennas is used to determinethe z-axis of the gestural input, as this mixturecorresponds directly to the performer’s overallproximity to the stand. The independent antenna’ssignal strengths correspond to each quadrant of thex–y space, so they are used to gather up/down andleft/right gestures, and visualized using a simpledisplay window in Max/MSP/Jitter.

Multimodal Detection Layer

The Multimodal Detection layer integrates audioand visual classification results (pitch and audio am-plitude estimates, face detections, angle estimates)as well as proximity of the performer to the electric-field sensors for gesture detection. A GUI allowscomposers to define the types of gestures occurringin the piece, either asynchronously (occurring atany time in the piece) or synchronously (orderedby timing). Gestures can be defined to occur in asingle modality alone (e.g., the occurrence of oneparticular note), or more robust combinations byrequiring that gestures occur in multiple modalitiestogether within a short time period. For example,a gaze to the side-mounted camera, along with acertain loudness of playing and/or proximity to oneantenna, can be required for a particular gesture.Upon detection of the pre-defined gesture, a triggeris sent to the synthesis machine in the form of anOSC message.

Audio Synthesis and Transformation

The sound synthesis component of the MMSS sys-tem is based on a client–server model (McCartney2002). The synthesis server receives network com-mands that are mapped into control logic and/ordirect modulation. The client–server model wasdeemed necessary to support distributed processingin cases where more complex analysis and synthesisalgorithms are required.

Overholt et al. 75

Figure 1. Multimodalinterface system model.

current system still lacks precise, dynamic synchro-nization of the acoustic instrument and computeralgorithm at smaller time scales. This problem willbe addressed in future versions of the MMSS.

MMSS as an Intelligent Space

Interactive music frequently addresses the issuesof interactivity by requiring performers to workwith additional physical elements to which theyare unaccustomed. These physical elements includefoot pedals (discrete and continuous), worn sensors,and devices attached to the instrument. These allowperformers to control computer-generated sounds,but they do not give performers the opportunity tointeract with the machine in the same way theyinteract with other performers. Many performersfind these elements to be distractions that interferewith their ability to perform (McNutt 2003).

To achieve more intuitive interactivity, an intel-ligent space is needed—a space that can sense manyaspects of the performer’s movements and infer theirintentions, thereby interacting accordingly. A spacesuch as this should be designed as a “person,” witheyes, ears, and a sense of dynamic three-dimensionalmotions. Of related interest is the MEDIATE

environment (Gumtau et al. 2005). MEDIATE is aresponsive environment replete with sensors (mi-crophones, interactive floor, cameras, interactivewall, and objects) that promotes multi-sensory in-teraction with the space. MEDIATE evaluates themeasurements of its sensors and makes decisionsabout the novelty or repetitiveness of participantactions to tailor media feedback accordingly.

The MMSS with its camera, microphone, andelectric-field sensing begins to embody the meta-concept of an intelligent, sensing, general perfor-mative space that can be expanded and furtherdeveloped for precise dynamic, interactive controlof any type of data. Whereas the current systemuses a standard video camera, microphone, andcustom electric-field sensing, future MMSS versionsmay incorporate three-dimensional “depth-of-field”cameras (e.g., the “Swiss Ranger” optical imagingsystem that provides real-time per-pixel distancedata at video frame rates), or directional microphonearrays, for example.

System Overview

Figure 1 describes the multimodal interface sys-tem model. The MMSS consists of three parts:

Overholt et al. 73

Born in 2008, Official release :Nov. 2009, IRCAM Forum ‣ used in concerts worldwide, 2 workshops, 10 actives pieces, ‣ active collaborations with composers

Score Following - Antescofo~

A. Cont « ANTESCOFO: Anticipatory Synchronization and Control of Interactive Parameters in Computer Music », International Computer Music Conference, North Irland, 2008

http://articles.ircam.fr/textes/Cont08a/



Score following• For Score Following References:

http://cosmal.ucsd.edu/arshia/index.php?n=Main.Scofobib

• http://imtr.ircam.fr/imtr/Score_Following_History

• Best systems use Markov/Semi-Markov modelling of musical events

performer

observation

position

training

Decodi

Observati

position in

Hidden Markov

scor



http://imtr.ircam.fr/imtr/Score_Following_History

http://imtr.ircam.fr/imtr/Score_Following_History


Score following

• Antescofo (Anticipatory Score Follower)

4

an occupancy distribution dj(u), where the random variableU accounts for the number of times spent in the macrostate j. Figure 2 shows a parametric macro-state Markovchain topology commonly used for duration modeling. Thisway, the macro-state consists of r Markov states and twofree parameters p and q corresponding respectively to theexit probability and the next-state transition probability. Themacro-state occupancy distribution associated to this generaltopology is the compound distribution:

P (U = u) =r!1!

n=1

"u! 1n! 1

#(1! p! q)u!nqn!1p

+"

u! 1r ! 1

#(1! p! q)u!rqr!1(p + q)

1 2 3 4 r

1-p-q 1-p-q 1-p-q 1-p-q 1-p-q

p+qq q q q

p

p

p

p...

q

Fig. 2. Parametric Markov Topology

If p = 0, this macro-state occupancy distribution is thenegative binomial distribution:

P (U = u) ="

u! 1r ! 1

#qr(1! q)u!r

which corresponds to a series of r states with no jumps tothe exit state with the shortcoming that the minimum timespent in the macro-state is r. This simplified version has beenwidely explored in various score following systems where thetwo parameters r and q are derived by optimization over themacro-state’s time duration provided by the music score ( [12],[13]).

2) Semi-Markov Time Occupancy: In a Semi-Markovmodel, a macro-state can be modelled by a single state (insteadof a fixed number of mico-states) and by using an explicittime occupancy probability distribution dj(u) for each statej and occupancy u. Assuming that Si is the discrete randomvariable denoting the macro-states at time i from a state spaceS " N, and Tm is the time spent at each state m, then St = mwhenever

m!

k=1

Tk # t <m+1!

k=1

Tk.

Or simply, we are at state m at time t when the duration mod-els for all states up to m and m+1 comply with this timing. Inthis configuration, the overall process is not a Markov processwithin macro-states but rather a Markov process in betweenmacro-states, hence the name semi-Markov.

The explicit occupancy distribution can then be defined asfollows:

dj(u) = P (St+u+1 $= j, St+u!v = j,

v % [0, u! 2]|St+1 = j, St $= j) (1)

where u = 1, . . . ,Mj with Mj the upper bound for the timespent in the macro-state.

Semi-Markov models were first introduced in [19] forspeech recognition and gained attention because of theirintuitive access to models’ temporal structure. Semi-Markovtopologies are usually much more sparse in computationsand controllable than their Markovian counterparts. Moreover,they provide explicit access to time models expressed asoccupancy distributions. Despite these advantages, explicitduration models might require substantial development ofstandard statistical inference algorithms. Such developmentscould become cumbersome if duration models are assumedto be dynamic (as is the case in our framework) rather thanstationary (as with most speech recognition problems).

III. GENERAL ARCHITECTURE

The general description of our score following task is asfollows: having possession of the symbolic music score inadvance, the goal of our system is to map the incomingrealtime audio stream onto this representation and decodethe current score position, realtime tempo and undertakescore actions. In this paper we focus on the first two (asdemonstrated through the example of figure 1). Score actionsinvolve musical programming either for live electronics effectsor automatic accompaniment applications and are reportedin [20]. The music score is represented by a probabilisticstate-space model constructed directly from a symbolic musicscore inspired by the observations in section II-B. Given thescore’s state-space representation, the realtime system extractsinstantaneous beliefs or observation probabilities of the audiofeatures calculated from the stream, with regard to the statesof the score graph. The goal of the system is to then integratethis instantaneous belief with past and future beliefs in orderto decode the position and tempo in realtime. Figure 3 showsa general diagram of our system.

Features

Inference & Decoding

Audio Tempo

Audio Stream

Score Position Tempo

Score Parser

Score

Score Actions

off-line

real-time

Fig. 3. General System Diagram

To tackle the problem, we adopt a generative approachwith the underlying hypothesis that the audio signal can begenerated by the underlying state-space score model. Formallyspeaking, we assume that the audio features through time !or x!

0 (short for x0, . . . , x! ) are stochastic processes repre-sented by the random variable {Xt}, which is generated by a

Arshia Cont. A coupled duration-focused architecture for realtime music to score alignment, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009 (in press).




Score Following / Gesture Follower

symbols

Modeling (HMM)

score

Modeling (HMM)

signal

gesture follower @ Ircamhttp://imtr.ircam.fr/imtr/Gesture_Follower

■ Bevilacqua, F., Zamborlin, B., Sypniewski, A., Schnell, N., Guédy, F., Rasamimanana, N. « Continuous realtime gesture following and recognition », accepted in Lecture Notes in Computer Science (LNCS), Gesture in Embodied Communication and Human-Computer Interaction, Springer Verlag. 2009

■ F. Bevilacqua, F. Guédy, N. Schnell, E. Fléty, N. Leroy, " Wireless sensor interface and gesture-follower for music pedagogy", Proc. of the International Conference of New Interfaces for Musical Expression (NIME 07), p 124-129, 2007

http://imtr.ircam.fr/imtr/Gesture_Follower

http://imtr.ircam.fr/imtr/Gesture_Follower

http://articles.ircam.fr/textes/Bevilacqua09b/




http://mediatheque.ircam.fr/articles/textes/Bevilacqua07a





Goals

• Hyp: Gesture « meaning » is in temporal evolutions

• Real-time gesture analysis :‣ gesture following: time progression of the

performed gesture‣ recognition/characterization: similarity of the

performed gesture to prerecorded gestures

• Requirements‣ simple learning procedure, with a single example‣ adaptation to the user idiosyncrasies‣ continuous analysis from the beginning of the gestures


Gesture ?

• Any continuous datastream of parameters• typically 0.1 to 1000 Hz• from motion capture systems: ‣ image descriptors‣ accelerometers, gyroscope, magnetometers

• from sound descriptors‣ pitch, loudness‣ mfccs, ...

• multimodal data


Time Profile Modeling: HMM

time

sensor value

probability density function

Markov Chains

transition probabilities

Markov Models


HMM structures

time time

one state every two samples

maximum relative speed = 2 maximum relative speed = 2

one state every sample


Hybrid Approach

• Hybrid between:‣ Template based - Dynamic Time Warping‣ Linear Dynamics Model‣ HMM

• Similar to S. Rajko et al. (ASU), also developed in an artistic context‣ G. Qian, T. Ingalls and J. James, Real-time Gesture Recognition with Minimal

Training Requirements and On-line Learning, to appear in IEEE Conference on Computer Vision and Pattern Recognition, 2007.

‣ S. Rajko and G. Qian, A Hybrid HMM/DPA Adaptive Gesture Recognition Method, ISVC 2005,p 227-234.

http://dancinghacker.com/research/papers/SemanticNetworkModel.pdf




http://dancinghacker.com/research/papers/HybridHmmDpaGestureRecognition.pdf




http://www.isvc05.net/

http://www.isvc05.net/


time

gesture parameter

recorded example

performed gesture (live)

Real-time time warping

• Synchronization/following

• Recognition

• Anticipation (prediction)


Time warping

time

acce

lera

tion

x

y

z

referencestime warped

performed gesture


Learning phase

• Transition matrix‣ left-to-right Markov chain‣ states regularly spaced in time⇒ transition matrix set by the sampling rate

⇒ direct relationship between state number i and time

(T= 1/1-a, where a is the self transition prob)

• Emission probabilities

values from the time profile

calculated or set by user


Forward Calculation

State probability for given

observation O(tn) = b

!

" tn+1( ) = A " t

n( ) # b[ ]

state probability

at t = tn+1

Transition Matrix


Decoding phase

• Using the forward computation [Rabiner 89] (causal !)

• Compute the probability α of being at state i

state i

!


Decoding phase

State with maximum probability at time t→ time progression

∑αi = likelihood at time t

state i

!


Evaluation with synthesized signals

!

test signalreference signal

!

noise!

offset!

scaling

mea

n er

ror

(sam

ple

#)


Following long sequences

!

Computation of α on a sliding window


Gesture Follower - Context

dance (performance

and installation)

music performance

music pedagogy

“gesture” = •acceleration•angular velocity•pressure•audio energy

StreicherKreis - Florence Baschet


Synchronizing Sound to Gesture


Music Pedagogy applications

• Conducting

Atelier des Feuillantines Fabrice Guédy


Homo Ludens (Richard Siegal - The Bakery)

2:55


Recognizing movement qualities

Sarah Fdili Alaoui (PhD work)Collaboration with the dance company Emio Greco I PC


Goal: classification / segmentation of sounds and gestures based on their temporal evolutionsApproach: segmental HMM models

400 600 800 1000 1200 1400 16001

2

3

4

5

6

7

8

steps

qt

yt

“classical” HMM

400 600 800 1000 1200 1400 16001

2

3

4

5

6

7

8

stepsGesture Follower

q1 q2 q3 qT

400 600 800 1000 1200 1400 16001

2

3

4

5

6

7

8

trajectories

lnqn

...

ytn!ln+1 ytn[ ]... ytn!1,, ,

segmental HMM

Towards Segmental Models


[1] J. Bloit, N. Rasamimanana, and F. Bevilacqua. Modeling and segmentation of audio descriptor profiles with segmental models. Pattern Recognition Letters, 2009.[2] J. Bloit, N. Rasamimanana, and F. Bevilacqua. Towards morphological sound description using segmental models. In DAFX, Como, Italy, 2009.

classification/segmentation on a violin database (pitch/loudness profiles)

Modelling by primitive assembling: Segmentation on a continuous stream:

10 11 12 13 14 15 16 17 180

0.2

0.4

0.6

0.8

1I4 I1 I2 I4

Loudness

time (s)

! " # # #$ # #sfz

#$ ##sfz

s1 s2 s3

s4 s5 s6

s7

s8 s9

f1

f2

f3

f4

f5

f6

Sound and gesture morphologies

PhD Julien Bloit & Projet Interlude

IRCAM- Real Time Musical Interactionstime

CA

DEN

gesture

Hierarchical / Two-level Modeling

1.Temporal Segments Temporal2. Sequence of Segments


Credits and Acknowledgements•Real Time Musical Interaction team:

Frédéric Bevilacqua, Tommaso Bianco, Julien Bloit, Riccardo Borghesi, Baptiste Caramiaux, Arshia Cont, Arnaud Dessein, Sarah Fdili Alaoui, Emmanuel Fléty, Vassilios-Fivos Maniatakos, Norbert Schnell, Diemo Schwarz, Fabrice Guédy, Alain Bonardi, Nicolas Rasamimanana, Bruno Zamborlin

•Current Support of the projects:

• ANR projects: Interlude, Topophonie (France).

• EU-ICT Project SAME

•Thanks toAtelier les Feuillantines and students, Remy Muller, Jean-Philippe Lambert, Alice Daquet, Anthony Sypniewski, Donald Glowinski, Bertha Bermudez and EG|PC, Myriam Gourfink, Richard Siegal, Hillary Goidell,Florent Berenger, Florence Baschet

Date post:	18-Mar-2018
Category:	Documents
Upload:	lekhuong
View:	217 times
Download:	4 times

Gesture Control of Music Systems - unifr.ch · PDF fileGesture Control of Music Systems...

Documents