Research on Music Perception and Action at Haskins Laboratories:

A Fast-Forward Review(Or: 25 Years in 8 Minutes)

Bruno H. Repp

Haskins Laboratories

270 Crown Street (1971-2005)

“Haskins Laboratories is an independent, international, multidisciplinary community of researchers conducting basic research on spoken and written language.” [What am I doing there?]

300 George Street

Beginnings: Melody and Text

Mary Lou Serafine (recent picture)

Robert G. Crowder (1939-2000)

Serafine, M. L., Crowder, R.G., & Repp, B. H. (1984). Integration of melody and text in memory for songs. Cognition, 16, 285-303. Serafine, M. L., Davidson, J., Crowder, R. G., & Repp, B. H. (1986). On the nature of melody-text integration in memory for songs. Journal of Memory and Language, 25, 123-135. Crowder, R. G., Serafine, M. L., & Repp, B. H. (1990). Physical interaction and association by contiguity in memory for the words and melodies of songs. Memory & Cognition, 18, 469-476.

Our research showed that in a test of memory for unfamiliar songs, melodies are more difficult to recognize if they are combined with different lyrics. A series of experiments explored the conditions under which this integration of melody and text occurs.

I started out as a speech researcher, working on speech perception and acoustic phonetics. However, in the 1980s I started a collaboration with Mary Lou Serafine and Robert Crowder on a musical topic of common interest.

The “Composer’s Pulse”

Manfred Clynes

Repp, B. H. (1989). Expressive microstructure in music: A preliminary perceptual assessment of four composers' "pulses". Music Perception, 6, 243-274.

Repp, B. H. (1990a). Composers' pulses: Science or art? Music Perception, 7, 423-434. Repp, B. H. (1990b). Further perceptual evaluations of pulse microstructure in computer performances of classical piano music. Music Perception, 8, 1-33. Repp, B. H. (1990c). Patterns of expressive timing in performances of a Beethoven minuet by nineteen famous pianists. Journal of the Acoustical Society of America, 88, 622-641.

At a conference in 1985, I encountered Manfred Clynes and his theory of composer-specific expressive microstructure, which intrigued me. With his help, I synthesized piano performances exhibiting “appropriate” and “inappropriate” microstructure, obtained aesthetic judgments from listeners, and analyzed the timing of expert pianists’ performances of a Beethoven sonata movement. The results were mixed.

Performance Analyses I

Repp, B. H. (1992b). Diversity and commonality in music performance: An analysis of timing microstructure in Schumann's "Träumerei". Journal of the Acoustical Society of America, 92, 2546-2568.

Repp, B. H. (1993b). Objective performance analysis as a tool for the musical detective. Journal of the Acoustical Society of America, 93, 1203-1204.

Repp, B. H. (1995c). Expressive timing in Schumann's "Träumerei": An analysis of performances by graduate student pianists. Journal of the Acoustical Society of America, 98, 2413-2427.

Repp, B. H. (1996b). The dynamics of expressive piano performance: Schumann's "Träumerei" revisited. Journal of the Acoustical Society of America, 100, 641-650.

Now I was hooked. I decided to abandon speech research and focus on music. Stimulated by my analysis of Beethoven performances, I conducted detailed measurements and statistical analyses of expert pianists’ expressive timing in performances of Robert Schumann’s “Träumerei”, op. 15, No. 7, which enabled me to give objective descriptions of commonalities and individual differences among famous artists’ interpretations. Later I analyzed both expressive timing and dynamics in MIDI recordings of graduate student pianists’ performances, which were demonstrably less individual than the expert performances.

Perception of Tempo and Timing

Repp, B. H. (1992c). A constraint on the expressive timing of a melodic gesture: Evidence from performance and aesthetic judgment. Music Perception, 10, 221-242.

Repp, B. H. (1994a). Relational invariance of expressive microstructure across global tempo changes in music performance: An exploratory study. Psychological Research, 56, 269-284.

Repp, B. H. (1994b). On determining the basic tempo of an expressive music performance. Psychology of Music, 22, 157-167.

Repp, B. H. (1995b). Quantitative effects of global tempo on expressive timing in music performance: Some perceptual evidence. Music Perception, 13, 39-57.

My “Träumerei” analyses led to several perceptual and performance studies using this music. They investigated the optimal shape of a ritardando, the scaling of expressive timing with changes in tempo, and the perceived tempo of an expressively timed performance.

Performance Analyses II

Repp, B. H. (1996c). Pedal timing and tempo in expressive piano performance: A preliminary investigation. Psychology of Music, 24, 199-221. Repp, B. H. (1996c). Patterns of note onset asynchronies in expressive piano performance. Journal of the Acoustical Society of America, 100,

3917-3932. Repp, B. H. (1996e). The art of inaccuracy: Why pianists' errors are difficult to hear. Music Perception, 14, 161-184.

Repp, B. H. (1997c). Some observations on pianists' timing of arpeggiated chords. Psychology of Music, 25, 133-148. Repp, B. H. (1997d). Expressive timing in a Debussy Prelude: A comparison of student and expert pianists. Musicae Scientiae, 1, 257-268.

Repp, B. H. (1997f). The effect of tempo on pedal timing in piano performance. Psychological Research, 60, 164-172.

Repp, B. H. (1997g). Variability of timing in expressive piano performance increases with interval duration. Psychonomic Bulletin & Review, 4, 530-534.

Making use of extensive MIDI performance data I had collected from Yale graduate student pianists, I measured and analyzed everything I could lay my hands on, including pedal timing and its dependence on tempo, arpeggio timing, asynchronies within and between hands, and error patterns. (Manfred Clynes called me “Reppmesser”.)

Piano Acoustics and Technique

Repp, B. H. (1993a). Some empirical observations on sound level properties of recorded piano tones. Journal of the Acoustical Society of America, 93, 1136-1144.

Repp, B. H. (1995a). Acoustics, perception, and production of legato articulation on a digital piano. Journal of the Acoustical Society of America, 97, 3862-3874.

Repp, B. H. (1997b). Acoustics, perception, and production of legato articulation on a computer-controlled grand piano. Journal of the Acoustical Society of America, 102, 1878-1890.

Repp, B. H. (1998f). Perception and production of staccato articulation on the piano. Unpublished manuscript.

During that time, I also conducted a few studies of piano acoustics, and of the production and perception of legato and staccato articulation on the piano, using simple musical materials.

Digression: The “Tritone Paradox”

Repp, B. H. (1994c). The tritone paradox and the pitch range of the speaking voice: A dubious connection. Music Perception, 12, 227-255.

Repp, B. H. (1997e). Spectral envelope and context effects in the tritone paradox. Perception, 26, 645-665.

Diana Deutsch

A stay in the Netherlands in 1993 enabled me to pursue research on the auditory illusion called “tritone paradox” and its presumable connection to language and speech, proposed by Diana Deutsch. My results did not settle the issue but led to some new paradoxes.

“Average” Performances

Repp, B. H. (1997a). The aesthetic quality of a quantitatively average music performance: Two preliminary experiments. Music Perception, 14, 419-444.

Repp, B. H. (1997d). Expressive timing in a Debussy Prelude: A comparison of student and expert pianists. Musicae Scientiae, 1, 257-268.

Inspired by recent findings in the psychological literature showing that composite faces constructed by averaging digitized images are judged to be more attractive than most individual faces, I synthesized composite music performances by averaging the timing and dynamic patterns of individual performances and found that they, too, were preferred to most individual performances (when realized on a digital piano). I also found that the average timing patterns of groups of expert and student pianists were remarkably similar, suggesting a common norm.

Action-Perception Parallels in Expressive Timing

Repp, B. H. (1992a). Probing the cognitive representation of musical time: Structural constraints on the perception of timing perturbations. Cognition, 44, 241-281.

Repp, B. H. (1995d). Detectability of duration and intensity increments in melody tones: A partial connection between music perception and performance. Perception & Psychophysics, 57, 1217-1232.

Repp, B. H. (1998a). Obligatory "expectations" of expressive timing induced by perception of musical structure. Psychological Research, 61, 33-43.

Repp, B. H. (1998c). Variations on a theme by Chopin: Relations between perception and production of deviations from isochrony in music. Journal of Experimental Psychology: Human Perception and Performance, 24, 791-811.

Repp, B. H. (1999g). Relationships between performance timing, perception of timing perturbations, and perceptual-motor synchronization in two Chopin preludes. Australian Journal of Psychology, 51, 188 203.

In a series of perceptual studies I demonstrated that the detectability of a small local change in the timing of an otherwise metronomic synthesized music performance is closely related to the average timing pattern of performances of the same music: If an interval is typically lengthened in performance, its artificial lengthening is difficult to detect, presumably because it sounds “normal”.

Performance Analyses III

Repp, B. H. (1998d). A microcosm of musical expression: I. Quantitative analysis of pianists' timing in the initial measures of Chopin's Etude in E major. Journal of the Acoustical Society of America, 104, 1085-1100.

Repp, B. H. (1999a). A microcosm of musical expression: II. Quantitative analysis of pianists' dynamics in the initial measures of Chopin's Etude in E major. Journal of the Acoustical Society of America, 105, 1972-1988.

Repp, B. H. (1999f). A microcosm of musical expression: III. Contributions of timing and dynamics to the aesthetic impression of pianists' performances of the initial measures of Chopin's Etude in E major. Journal of the Acoustical Society of America, 106, 469-478.

In my most ambitious performance analysis project, I measured the timing of 115 recorded performances of the opening phrase of Chopin’s Etude in E major, Op. 10, No. 3, and statistically extracted four underlying independent timing patterns whose weighted combinations yielded good approximations of the observed patterns. I also analyzed the measured dynamic patterns and investigated the (very weak) relation between these objective measures and aesthetic judgments of the performances.

More Chopin Etude Studies

Repp, B. H. (1998b). The detectability of local deviations from a typical expressive timing pattern. Music Perception, 15, 265-290.

Repp, B. H. (1999b). Detecting deviations from metronomic timing in music: Effects of perceptual structure on the mental timekeeper. Perception & Psychophysics, 61, 529-548.

Repp, B. H. (1999c). Control of expressive and metronomic timing in pianists. Journal of Motor Behavior, 31, 145-164.

Repp, B. H. (1999d). Effects of auditory feedback deprivation on expressive piano performance. Music Perception, 16, 409-438. Repp, B. H. (2000). Pattern typicality and dimensional interactions in pianists’ imitation of expressive timing and dynamics. Music

Perception, 18, 173–211.

Repp, B. H. (2002a). The embodiment of musical structure: Effects of musical context on sensorimotor synchronization with complex timing patterns. In W. Prinz & B. Hommel (Eds.), Common mechanisms in perception and action: Attention and Performance XIX (pp. 245–265). Oxford, U.K.: Oxford University Press.

Repp, B. H. (2002e). Perception of timing is more context sensitive than sensorimotor synchronization. Perception & Psychophysics, 64, 703–716.

The Chopin performance data enabled me to conduct other studies that relied on them, including experiments on performance imitation, the role of auditory feedback, mental imagery, and pattern learning.

First Tapping Studies

Repp, B. H. (2000a). Compensation for subliminal timing perturbations in perceptual-motor synchronization. Psychological Research, 63, 106–128.

Repp, B. H. (2001a). Phase correction, phase resetting, and phase shifts after subliminal timing perturbations in sensorimotor synchronization. Journal of Experimental Psychology: Human Perception and Performance, 27, 600–621.

Repp, B. H. (2002b). Automaticity and voluntary control of phase correction following event onset shifts in sensorimotor

synchronization. Journal of Experimental Psychology: Human Perception and Performance, 28, 410–430. Repp, B. H. (2002c). Phase correction in sensorimotor synchronization: Nonlinearities in voluntary and involuntary responses to

perturbations. Human Movement Science, 21, 1–37.

Repp, B. H. (2002d). Phase correction following a perturbation in sensorimotor synchronization depends on sensory information. Journal of Motor Behavior, 34, 291–298.

In some of my recent music studies I had used finger tapping as a method of tracking expectations about expressive timing. I discovered that small timing changes in the music that were barely audible were automatically compensated for in tapping. This awakened my interest in the phase correction process underlying sensorimotor synchronization. I confirmed my finding in several studies using phase perturbation methods. From now on , my research focused on simple rhythms and only rarely used real music.

Phase and Period Correction

Repp, B. H. (2001b). Processes underlying adaptation to tempo changes in sensorimotor synchronization. Human Movement Science, 20, 277–312.

Repp, B. H., & Keller, P. E. (2004). Adaptation to tempo changes in sensorimotor synchronization: Effects of intention, attention, and awareness. Quarterly Journal of Experimental Psychology, 57A, 499–521.

Peter E. Keller

Together with Peter Keller, who spent one year with me as a post-doc, I investigated how synchronized tapping adapts to tempo change in a metronome (i.e., period correction). We found period correction to be less automatic and more cognitively controlled than phase correction. Peter (now at the Max Planck Institute for Human Cognitive and Brain Sciences in Leipzig, Germany) has remained a close collaborator.

Synchronization in Two Modalities

Repp, B. H., & Penel, A. (2002). Auditory dominance in temporal processing: New evidence from synchronization with simultaneous visual and auditory sequences. Journal of Experimental Psychology: Human Perception and Performance, 28, 1085–1099.

Chen, Y., Repp, B. H., & Patel, A. D. (2002). Spectral decomposition of variability in synchronization and continuation tapping:

Comparisons between auditory and visual pacing and feedback conditions. Human Movement Science, 21, 515–532.

Repp, B. H., & Penel, A. (2004). Rhythmic movement is attracted more strongly to auditory than to visual rhythms. Psychological Research, 68, 252–270.

Patel, A. D., Iversen, J. R., Chen, Y., & Repp, B. H. (2005). The influence of metricality and modality on synchronization with a beat. Experimental Brain Research, 163, 226–238.

Amandine Penel, another post-doc, and I found that people had difficulty synchronizing with visual sequences, especially in the presence of auditory distractors. I also started collaborating with Aniruddh Patel and John Iversen (Neurosciences Institute, San Diego). We found that it is difficult to extract a beat from a visual rhythm.

Aniruddh D. Patel John R. Iversen

Distractors and Syncopation

Repp, B. H. (2003a). Phase attraction in sensorimotor synchronization with auditory sequences: Effects of single and periodic distractors on synchronization accuracy. Journal of Experimental Psychology: Human Perception and Performance, 29, 290–309.

Keller, P. E., & Repp, B. H. (2004). When two limbs are weaker than one: Sensorimotor syncopation with alternating hands. Quarterly Journal of Experimental Psychology, 57A, 1085-1101.

Repp, B. H. (2004b). On the nature of phase attraction in sensorimotor synchronization with interleaved auditory sequences.

Human Movement Science, 23, 389–413.

Keller, P. E., & Repp, B. H. (2005). Staying offbeat: Sensorimotor syncopation with structured and unstructured auditory sequences. Psychological Research, 69, 292–309.

Repp, B. H. (2006a). Does an auditory distractor sequence affect self-paced tapping? Acta Psychologica, 121, 81–107.

In a series of experiments, I investigated the influence of auditory distractors on synchronization with an auditory metronome or on self-paced tapping. Peter Keller took the lead in investigations of off-beat synchronization with alternating hands, a surprisingly difficult task.

The Synchronization Threshold

Repp, B. H. (2003b). Rate limits in sensorimotor synchronization with auditory and visual sequences: The synchronization threshold and the benefits and costs of interval subdivision. Journal of Motor Behavior, 35, 355–370.

Repp, B. H. (2005a). Rate limits of on-beat and off-beat tapping with simple auditory rhythms: 1. Qualitative observations. Music Perception, 22, 479–496.

Repp, B. H. (2005b). Rate limits of on-beat and off-beat tapping with simple auditory rhythms: 2. The role of different kinds of accent. Music Perception, 23, 167–189.

Repp, B. H. (2007a). Hearing a melody in different ways: Multistability of metrical interpretation, reflected in rate limits of sensorimotor synchronization. Cognition, 102, 434-454.

Repp, B. H. (2007b). Perceiving the numerosity of rapidly occurring auditory events in metrical and non-metrical contexts.

Perception & Psychophysics, 69, 529-543.

In several studies, I determined the fastest rate at which musicians are able to maintain synchrony with a metronome or a simple rhythm. This “synchronization threshold” reflects a sensorimotor processing limit around 8-10 Hz.

Review Papers

Repp, B. H. (2005c). Sensorimotor synchronization: A review of the tapping literature. Psychonomic Bulletin & Review, 12, 969 –992.

Repp, B. H. (2006c). Musical synchronization. In E. Altenmüller, M. Wiesendanger, & J. Kesselring (Eds.), Music, motor control, and the brain (pp. 55–76). Oxford, UK: Oxford University Press.

Repp, B. H. (2006d). Rate limits of sensorimotor synchronization. Advances in Cognitive Psychology (http://ac-psych.org/), 2, 163-

181.

Halfway through the decade I published three review papers.

Perception and Action: Pianists

Repp, B. H., & Knoblich, G. (2004). Perceiving action identity: How pianists recognize their own performances. Psychological Science, 15, 604–609.

Keller, P. E., Knoblich, G., & Repp, B. H. (2007). Pianists duet better when they play with themselves: On the possible role of

action simulation in synchronization. Consciousness and Cognition, 16, 102-111. Repp, B. H., & Keller, P. E. (2010). Self versus other in piano performance: Detectability of timing perturbations depends on

personal playing style. Experimental Brain Research, 202, 101-110.

Günther Knoblich

In 2003 I began regular visits to the Max Planck Institute for Human Cognitive and Brain Sciences in Munich, later in Leipzig. Günther Knoblich (now at Radboud University Nijmegen) taught me much about perception and action and later invited me to work with him for two years at Rutgers University, Newark, when I did not have any grant support.

In three studies involving real music, we showed that pianists can recognize their own performances and predict their own characteristic timing.

Perception and Action: Tapping and Listening

Repp, B. H. (2006b). Does an auditory perceptual illusion affect on-line auditory action control? The case of (de)accentuation and synchronization. Experimental Brain Research, 168, 493–504.

Repp, B. H., & Knoblich, G. (2007a). Action can affect auditory perception. Psychological Science, 18, 6-7.

Repp, B. H., & Knoblich, G. (2007b). Toward a psychophysics of agency: Detecting gain and loss of control over auditory action effects. Journal of Experimental Psychology: Human Perception and Performance, 33, 469-482.

Knoblich, G., & Repp, B. H. (2009). Inferring agency from sound. Cognition, 111, 248-262.

Repp, B. H., & Knoblich, G. (2009). Performed or observed keyboard actions affect pianists’ judgments of relative pitch. Quarterly Journal of Experimental Psychology, 62, 2156-2170.

In other studies, we investigated the feeling of agency in tapping and showed that pressing piano keys can affect pianists’ perception of relative pitch in the tritone paradox.

Rhythm Production and Perception

Repp, B. H., Windsor, L., & Desain, P. (2002). Effects of tempo on the timing of simple musical rhythms. Music Perception, 19, 565–593.

Repp, B. H., & Saltzman, E. L. (2002). Influences of metrical structure and grouping on the kinematics of rhythmic finger tapping. Unpublished manuscript.

Repp, B. H., London, J., & Keller, P. E. (2005). Production and synchronization of uneven rhythms at fast tempi. Music Perception, 23, 61–78.

Repp, B. H. (2008a). Multiple temporal references in sensorimotor synchronization with metrical auditory sequences. Psychological Research, 72, 79-98.

Repp, B. H., Iversen, J. R., & Patel, A. D. (2008). Tracking an imposed beat within a metrical grid. Music Perception, 26, 1-18.

Repp, B. H., London, J., & Keller, P. E. (2008). Phase correction in sensorimotor synchronization with nonisochronous sequences. Music Perception, 26, 171-175.

Repp, B. H., London, J., & Keller, P. E. (submitted). Perception-production relationships and phase correction in synchronization with two-interval rhythms.

With various collaborators, I investigated effects of tempo, meter, and interval structure on the accuracy of rhythm production and perception, beat tracking, and phase correction in synchronization with moderately complex rhythms.

Inter-Agent Synchronization

Keller, P. E., & Repp, B. H. (2008). Multilevel coordination stability: Integrated goal representations in simultaneous intra-personal and inter-agent coordination. Acta Psychologica, 128, 378-386.

Repp, B. H., & Keller, P. E. (2008). Sensorimotor synchronization with adaptively timed sequences. Human Movement Science, 27, 423-456.

Synchronization between individuals (as in music performance) has as yet been little investigated but is a special interest of Peter Keller. I have done some preliminary work with him along these lines.

Yale Undergraduate Collaborators

Resnicow, J. E., Salovey, P., & Repp, B. H. (2004). Is recognition of emotion in music performance an aspect of emotional intelligence? Music Perception, 22, 145–158.

Repp, B. H., & Doggett, R. (2007). Tapping to a very slow beat: A comparison of musicians and non-musicians. Music Perception, 24, 367-376.

Repp, B. H., & Bruttomesso, M. (2009). A filled duration illusion in music: Effects of metrical subdivision on the perception and production of beat tempo. Advances in Cognitive Psychology, 5, 114-134.

Repp, B. H., & Jendoubi, H. (2009). Flexibility of temporal expectations for triple subdivision of a beat. Advances in Cognitive

Psychology, 5, 27-41.

Repp, B. H., & Steinman, S. R. (2010). Simultaneous event-based and emergent timing: synchronization, continuation, and phase correction. Journal of Motor Behavior, 42, 111-126.

Repp, B. H., & Marcus, R. J. (in press). No sustained sound illusion in rhythmic sequences. Music Perception. Repp, B. H., & Thompson, J. M. (in press). Context sensitivity and invariance in perception of octave-ambiguous tones.

Psychological Research.

In recent years, I have been sought out by an increasing number of Yale undergraduates who wish to do research on music cognition. This has resulted in a number of coauthored publications on various topics.

Current Research

Repp, B. H. (2008b). Metrical subdivision results in subjective slowing of the beat. Music Perception, 26, 19-39.

Repp, B. H. (2008c). Perfect phase correction in synchronization with slow auditory sequences. Journal of Motor Behavior, 40, 363-367.

Repp, B. H. (2009a). Segregated in perception, integrated for action: Immunity of rhythmic sensorimotor coordination to auditory stream segregation. Quarterly Journal of Experimental Psychology, 62, 426-434.

Repp, B. H. (2009b). Rhythmic sensorimotor coordination is resistant but not immune to auditory stream segregation. Quarterly

Journal of Experimental Psychology, 62, 2306-2312. Iversen, J. R., Repp, B. H., & Patel, A. D. (2009). Top-down control of rhythm perception modulates early auditory responses.

Annals of the New York Academy of Sciences, 1169, 58-73. Repp, B. H. (2010). Sensorimotor synchronization and perception of timing: Effects of music training and task experience. Human

Movement Science, 29, 200-213.

Repp, B. H. (in press). Self-generated interval subdivision reduces variability of synchronization with a very slow metronome. Music Perception.

Repp, B. H. (in press). Do metrical accents create illusory phenomenal accents? Attention, Perception, & Psychophysics.

I am continuing to investigate sensorimotor synchronization in various contexts, as well as perception of rhythm, meter, and accentuation.

Thanks to:

• Haskins Laboratories, for allowing me to stay even though my research is unrelated to the Laboratories’ mission

• NIH and NSF for providing financial support

• The “Master Tappers” (my regular participants: nowadays graduate students from the Yale School of Music, previously also including undergraduates and other graduate students), for their cheerful cooperation and excellent rhythmic skills

• You, the audience, for your attention