ABSTRACTPiano&Dancer is an interactive piece for a dancer and an electrome-chanical acoustic piano. �e piece presents the dancer and thepiano as two performers on stage whose bodily movements aremutually interdependent. �is interdependence reveals a close re-lationship between physical and musical gestures. Accordingly,the realisation of the piece has been based on creative processesthat merge choreographic and compositional methods. In orderto relate the expressive movement qualities of a dancer to the cre-ation of musical material, the piece employs a variety of techniques.�ese include methods for movement tracking and feature analy-sis, generative algorithms for creating musical structures, and theapplication of non-conventional scales and chord transformationsto shape the modal characteristics of the music. �e publicationcontextualises Piano&Dancer by relating its creation to concepts ofembodiment, interactivity and musical structure and by discussingopportunities for creative cross-fertilisation between dance chore-ography and musical composition. It also provides some detailsabout the challenges and potentials of integrating a mechanical mu-sical instrument into an interactive se�ing for a dance performance.Finally, the paper highlights some of the technical and aestheticprinciples that were used in order to connect expressive qualitiesof body movements to the creation of music structures.
CCS CONCEPTS•Human-centered computing → Interaction design theory,concepts and paradigms; Interaction design process and methods;Gestural input; •Applied computing → Performing arts; Soundand music computing;
KEYWORDSDance Technology, Interactive Soni�cation, Music and Movement,Gesture-based Interaction, Embodiment, Automated Analysis ofMovement �alities
∗Produces the permission block, and copyright information†�e full version of the author’s guide is available as acmart.pdf document
ACM Reference format:Pablo Palacio and Daniel Bisig. 2017. Piano&Dancer - Interaction Betweena Dancer and an Acoustic Instrument. In Proceedings of MOCO’17, London,United Kingdom, 28-30 June, 2017, 8 pages.DOI: h�p://dx.doi.org/10.1145/3077981.3078052
1 INTRODUCTIONPiano&Dancer is a dance piece for a single human dancer and anelectromechanical acoustic piano (Disklavier). �is piece has beenrealised as a collaboration between the two authors of this publi-cation and the choreographer and dancer Muriel Romero. Duringthe performance, both the dancer and the Disklavier are presenton stage. �e music of the performance is produced through thepiano’s mechanical movements. �ese movements are generated inreal-time through a combination of compositional algorithms, sto-chastic functions, swarm simulations, and mappings according tospeci�c musical modes, all of which are in�uenced by the dancer’sbodily movements. As a result, the piano and dancer are connectedwith each other through three layers of relationships: they are bothphysically present on stage and exhibit bodily movements, theirrespective movements are correlated through an algorithmic inter-mediate layer, and they exhibit in the musical and bodily domain astrong correspondence in expressivity.
One of the central motivations for the realisation of this piece isbased on the assumption that a pianist’s bodily movements duringa musical performance represents a choreography. �is chore-ography is at the same time very sophisticated but also highlyconstrained. Furthermore, the experiential aspects of this chore-ography are revealed through the resulting musical forms. �eseforms are shaped by the intrinsic physical properties of the pianoand by the characteristics of the gestures performed by the player.
Starting from these considerations, the piece Piano&Dancer un-dertakes an artistic investigation into the choreographic qualitiesof instrumental gestures and how these qualities can be creativelyextended while at the same time maintaining their inherent musicproducing functionality. �ese investigations form part of a largerresearch context concerning the functional and aesthetic relation-ships between body gestures and musical gestures. Some researchquestions within this context are: Can musical structures conveyacoustically some of the qualitative and expressive characteristicsof bodily movement? Which intimate aspects of a dancer’s bodilyactivities that are normally hidden to an external observer can beconveyed through acoustic feedback? What formal and aestheticmechanisms need to be established to mediate between the bodily
MOCO’17, 28-30 June, 2017, London, United Kingdom P. Palacio and D. Bisig
and musical domains of gestural movements? How can the im-portant role of kinaesthetic body awareness be taken advantageof for the control and perception of interactive music? How cancompositional and choreographic approaches inform each other inthe creation of performance pieces for dancing instrumentalists?
For the realisation of Piano&Dancer, an interactive se�ing waschosen that abolishes the necessity for a direct tactile manipulationof the acoustic instrument and thereby relaxes the traditional func-tional constrains of piano gestures. In addition, a direct causalitybetween the physical aspects of the dancer’s body movements andpiano actuation was avoided in favour of interaction techniquesthat take the expressive aspects of bodily movement into accountand that integrate compositional algorithms as part of their map-ping mechanisms. Some of the technology that has been employedwas developed in the context of the European H2020 ICT researchprojects DANCE and WHOLODANCE. �e DANCE project studieshow music can convey qualities of body movement and enablesa cross-modal transfer of movement perception into the acousticdomain. �e WHOLODANCE project develops and applies tech-nologies that provide educational tools for dance practitioners.
2 BACKGROUND�is section provides a condensed overview about some of thetopics that are of relevance for this project. �ese topics include:the development of design principles for interactive music systemsthat draw from embodied cognition research, the mutual cross-fertilisation between creative approaches in dance and music, thecomputational analysis of higher level movement features fromraw sensory data, and the correlation of movement features withthe articulation of sonic gestures. Other relevant topics such asthe combination of machines and performers on stage and theapplication of algorithmic and generative approaches in musiccomposition have been discussed in a previous publication [6].
Music in its origins is inseparable from the manipulation of physicalobjects with our body in order to produce sound. �e coordinationof movements that are necessary to control an acoustic instrumentcorresponds in itself to a very sophisticated choreography thatrequires a thorough awareness of body position and body motion,both of which have large e�ects on the resulting sounds. �eoriesfrom the �eld of embodied cognition emphasise the importantrole of bodily experience and engagement with the world for thedevelopment of cognitive capabilities.
�e notion of image schemas refers to cognitive representationsof the intimate and preconscious relationships between recurringpa�erns in perception and action that occur when we interact withand move through our environments [16]. It is assumed that manyof these representation form the foundations for higher level cog-nitive capabilities. Exploratory and intuitive forms of engagementwith interactive systems very likely are guided by image schemas[7, 14]. Similarly, certain key musical principles that inform theorganisation of sound arise from a preconscious understanding ofour body and environment [17]. For example, the perception ofrhythms is based on a disparity of durations, intensities and pitches
which in order to be recognised requires at least two sonic eventsto occur. As in human walking, the �rst event is an elevation andthe second is a fall. In musical terms, this corresponds to an upbeatand a downbeat, and a tension and a release [21]. Musical princi-ples emerging from image schema related to verticality are alsovery prevailing. Moreover, as Graham and Bridges explain in [13],these image schemas can be considered to be closely linked to thearchetypal sonic dynamic morphologies that Smalley describes inhis prominent theory of Spectromorphology [29]. �is relationshipis relevant in the context of Piano&Dancer, since the sound motionsportrayed by Smalley such as �ow, push/drag, rise, �oat all haveclear connotations in the bodily domain.
Aspects of Laban analytic notation [23] that serve to describethe stress pa�erns and internal �ow of body movements can alsobe related to the articulation of music and sound. In traditionalwestern music notation, these qualitative aspects are concernedmainly with how the sounds are connected or stressed, the legatoand staccato and how these two are combined. �ese aspects arealso important in the articulation of spoken language. Here, thestress pa�erns and other components belong to what is de�nedas paralanguage and give rise to expressive aspects of speech thatseem to be independent of verbal content [3]. In a similar way, amovement quality carries information about the dancer’s expres-sivity and may possess a degree of independence that allows it tobe imposed on di�erent movement trajectories and thereby alterits perceived characteristics [15]. Although the Laban frameworkhas been used as a way to conceptualise movement in an abstractmanner, it has also enriched the study of the phenomenologicalaspects embodiment. �e Laban framework provides a complemen-tary point of view from cognitive theories in that is emphasises theexperiential aspects of bodily awareness and their role in buildingup personal experience [18]. As Levishon and Schiphorst state intheir contribution, human computer interaction design may bene�tfrom the study of how movement is experienced in �rst person,concentrating on the qualitative and expressive nature of move-ment. �is approach is connected to dance studies such as somaticsand complements the image schema model.
�ese previous considerations help us to understand how metaphor-ical relationships between embodied cognition principles and theanalysis of natural morphologies of sound can inform the designof mapping strategies for interactive music. Furthermore, theseconsiderations can lead over to inform creative strategies for con-verging and complementing aesthetic principles in dance and music.And more speci�cally, they provide inspirations for how the qual-itative aspects of bodily articulations can be transformed into anarticulation of sound objects [32].
2.2 Cross-Fertilisation Between Movement andSonic Arts
One of the sources of inspiration for Piano&Dancer is the assump-tion of the existence of common principles underlying dance andmusic. Probably the most ancient but very evident fact of this re-lationship can be traced back to the 4500 BC. A technique namedChironomy [1], a term deriving from the Greek word quiro whichtranslates to hand, illustrates how music was created in the ancientEgypt empire. In this technique, a system of gestures involving
Piano&Dancer - Interaction Between a Dancer and an Acoustic Instrument MOCO’17, 28-30 June, 2017, London, United Kingdom
the posture and spatial trajectories of the hand, �ngers, arms, aswell as the choice of le� and right, were used to control every mu-sical detail performed by one or several instrument players. �evertical spatial trajectories of these singing hands may be regardedas the basis for western music notation systems [24]. �is veryold technique has served as an inspiration for the developmentof one of the scenes in Piano&Dancer during which the dancersperforms Chironomy-based gestures that are translated in real timeinto music gestures by the piano (see Figure 1).
Figure 1: Performance Scene that is Inspired by the AncientEgyptian Chironomy Technique
Other spatial techniques of choreographic nature have inspiredmusic composition throughout history and specially during thetwentieth century. One of the most evident examples is how thesonic spatialisation techniques in electroacustic music have intro-duced the geometrical space as a compositional dimension. Anoverview of spatialisation approaches in musical composition isprovided in the PhD thesis by Bates [2]. Wishart’s seminal text OnSonic Art [32] provides a thorough description of spatial motionsfor sonic objects. Some of these motions could be interpreted aschoreographic cues. Terms employed to describe these motionsare: direct, cyclical, symmetric, looping, zig-zag, forward, backward,elastic, bounced, frame rotations, accelerating, and many more.
Also relevant in the context of this publication are gesture-basedmusic interfaces that have been speci�cally designed to allow un-constrained motions. Contrary to more conventional interfaces,these gesture-based interfaces o�er musicians a freedom of move-ment and dexterous control which is not unlike that of dancers[25]. A good example of this approach are non-tactile interfaces.For the successful use of such interfaces, an advanced sense ofproprioception and kinaesthetic awareness needs to be acquiredby the musician [27]. Accordingly, the study of dancers might pro-vide important inspirations for musicians who plan to design andcontrol these type of interfaces [19]. An example of an interfacethat serves the speci�c purpose of translating dance into music isthe Motion Composer [4]. �is interface integrates a camera-basedmotion tracking system and is meant to aid people with physical dis-abilities to improve the movement awareness. Another exemplaryinterface is Coniglio’s MidiDancer [11]. �is interface consists of asensor equipped suit that allows the dancer to control music. �eintended use of this interface is to encourage dancers to move likemusicians in order to interact with the musical system.
In Piano&Dancer, the interactive relationship between dancerand electromechanical piano is based on the concept of a non-tactileexchange of mechanical energy, through which the dancer’s move-ments control the actuators that set into motion the piano hammers.By allowing the performer to control the piano through other meansthan direct tangible interaction, the functional constraints of soundproducing gestures and the immediacy of their e�ects on the mu-sical result are dissolved. �is provides the opportunity to inventnovel and diversi�ed relationships between physical and musicalgestures. As a result, the dancer’s bodily movements can be shapedaccording to choreographic criteria. Furthermore, through algo-rithmic means, the relationship between movement and music canbe expanded to involve interactive control over the compositionalprocess itself.
�e transfer of music compositional techniques into choreo-graphic principles is also not uncommon in contemporary dance.A movement or a melody are objects extended in time that canbe transformed through a variety of means. For example, the tra-ditional transformations of a melodic pa�ern, its inversion, itsretrograde and retrograde inversion have become movement gen-eration strategies for William Forsythe and other choreographersfollowing his example [10] . It is important to mention in this con-text how these popular melodic transformation techniques haveactually originated from the spatial domain, since they are derivedfrom symmetric group transformations of the rectangle. �ere ex-ist many more transformation possibilities since a melodic line ora movement can be rotated through any angle, as Forsythe hasshowed choreographically in [10] or Xenakis in the musical domainwith his Arborescences theory [30]. Another popular technique ismusical counterpoint. In music, this term refers to the establish-ment of harmonic relationships between two melodic lines whoserhythm and contour are di�erent. In dance, the counterpoint princi-ple can be applied to align and merge choreographed sections intocoherent unities that di�er in their use of space, time, or movement.�is technique has been extensively used by choreographers suchas William Forsythe, Marius Petipa, George Balanchine, TrishaBrown, Jonathan Burroughs, and Pablo Ventura [31]. �e adoptionof these musical principles as a basis for developing choreographicstructures has played an important role in the merging of choreo-graphic and music compositional methods that guided the creativeprocess of Piano&Dancer.
3 REALISATIONIn this section, we describe the technical implementations thatform part of the realisation of Piano&Dancer. A major focus isplaced on the developments that enable the dancer to control thepiano without direct tangible interaction. �is remote mode ofinteraction with an acoustic piano is based on an analysis of certainqualities of the dancer’s movements which serve as input for thegeneration and control of the sounding material. Accordingly, wedescribe in some detail which aspects of the dancer’s movement areextracted, how these aspects are mapped to control the mechanicalmovements of the piano, and how the resulting sounds re�ect andcombine in their properties both the extrinsic aspects of the dancer’smovements and the intrinsic properties of the piano’s sound pro-ducing mechanism. �e real time data we gather from the dancer’s
MOCO’17, 28-30 June, 2017, London, United Kingdom P. Palacio and D. Bisig
movements can be organised in two layers [9]. �e �rst layer cor-responds to the physical aspects of movement which is directlymeasured by the sensing devices (rotational accelerations, positionof joints, etc.). �e second layer deals with higher level propertiesof movement that can be obtained by analysing the raw sensoryinput. Movement properties from both the �rst and second layerare then further processed by an additional layer which comprisesthe generative algorithms such as stochastic distributions or swarmsimulations that underly the automated composition of the music.As a result of these relationships, the choreographic and musiccompositional principles are tightly interconnected. �e composi-tion of Piano&Dancer emerges from the integration of movementgestures and corresponding sonic gestures that mirror the dancer’sexpressivity. A more detailed description of the choreographic andmusical aspects of Piano&Dancer can be found in [6].
3.1 Initial Considerations and ChallengesOne of the challenges of developing this work concerns the es-tablishment of relationships between the aesthetic principles of adancing human body and that of a mechanical piano. �e dancerexhibits a high degree of freedom and continuity in her movements.�e piano on the other hand represents a highly constrained musi-cal mechanism whose properties have been optimised to match theformalised principles of western notated music. Accordingly, weface the challenge of how to transfer a gestural activity that evolvesin a multidimensional continuum into another gestural activitythat is con�ned into a one-dimensional and discrete la�ice. Anadditional challenge concerns the fact that the dancer lacks hapticfeedback from the instrument that are regular pianist experiences.As Gillespie explains in [12], the hearing sense informs the per-former of the acoustical behaviour of an instrument, but hapticfeedback conveys information about the instrument’s mechanicalbehaviour. �e perception of this mechanical behaviour plays aquite important role in the process of learning and playing an in-strument. As compensation mechanism, kineasthetic awarenessbecomes an extremely important experiential aspect when interact-ing with musical systems that don’t provide haptic feedback [19]. Insuch situations, kineasthetic feedback provides an important meansof assessment for training the motor control needed to master thesystem. However, compared to haptic feedback, kinaesthetic aware-ness plays an inferior role for musicians in that it is less suitablefor achieving accurate and repeatable musical results[27].
�e other side of this relationship is also of crucial importance,namely, how or to what degree does the morphology of a soundingobject depend on the morphology of the performer’s kinetic energy.To address this topic, it is useful to iterate that every sound-eventpossesses an intrinsic and an extrinsic morphology [32]. �e in-trinsic morphology of a sound is shaped by the sound producingmechanisms of an acoustic instrument. Apart from the fact thatthe sound has to be triggered through a singular impulse by theplayer, its intrinsic properties are independent from the performer.Sounds whose extrinsic morphology is dominant are producedthrough a continued excitation through which the morphology ofthe sound producing activity of the human performer is imprinted.Sounds that possess predominately an intrinsic morphology areproduced by instruments such as the piano, or by instruments that
are plucked or percussive. Sounds that possess predominately anextrinsic morphology are produced by instruments that are blowedor bowed. �e consequence from this observation is that once adance movement sets into motion a piano key, the performer has nocontrol on the morphology of the resulting discrete sound. Contin-uous modulation is only possible in a very limited manner throughthe control of the damping and mute pedals.
However, an alternative means to provide the dancer with aseemingly continuous control over the extrinsic morphology ofa piano sound is based on the creation of a acoustic stream inwhich discrete note events fuse into a single perceptual unit. �isis the case for iterative and continuous sound morphologies suchas fast trills, tremolos or glissandi, in which the adjacent notes thatcomposed the temporal sequence are no longer distinguishable asindividual sonic events but rather become perceptually merged. Aswe will see below such modes of sound continuation can be usedas the sonic substrate that can be shaped by certain qualitativesaspects of movement.
3.2 Movement Sensing and Feature AnalysisFor almost all the scenes in Piano&Dancer, interactivity is based onsensing the dancer’s movements with inertial measurement units(IMU) that are a�ached to the dancer’s body. �is technique iscomplemented only during particular moments by a camera-basedtracking system. �e camera based tracking systems provides anallocentric and absolute frame of reference whereas IMU sensorsprovide an egocentric and therefore relative frame of reference.�e camera system is used to enable the dancer to set into motionthe piano keys or to accentuate notes based on the spatial posi-tion of her joints and regardless of whether an IMU is a�achedto that particular joint or not. �e IMU devices that are used forthe piece are named Xosc and are provided by the company x-ioTechnologies. �ese devices integrate a gyroscope, an accelerom-eter and a magnetometer, each of them providing three degreesof freedom. Furthermore, these devices o�er Wi�-based wirelessconnectivity whose bandwidth and latency is excellent even whenmultiple IMU devices are used concurrently. For the piece, four ofthese devices are employed to track the movements of four jointson the dancer’s body (two wrists and two ankles). Interactivity isbased on both the acquisition of raw sensory data as well as onthe analysis of higher level movement features such as Smoothness,Weight, Energy and Dynamic Symmetry [9]. �e characteristics ofthese movement features are inspired by the de�nitions proposed bychoreographer Rudolf Laban [23] but occasionally deviate in theirexact implementation from these de�nitions. By integrating meth-ods for higher level feature analysis into an interactive system, thissystem becomes capable of detecting and subsequently respondingto movement qualities that are also salient for the dancer and thehuman audience [28]. �is helps to alleviate one of the problemswhen applying interactive technology for dance: the constrainingof dance movements through technological prerequisites and theshi�ing of the dancer’s a�ention away from intentionality and ex-pressivity towards the purely functional execution of movement[20].
�e analysis of higher level movement features is implementedin the EyesWeb programming environment. As part of two EU
Piano&Dancer - Interaction Between a Dancer and an Acoustic Instrument MOCO’17, 28-30 June, 2017, London, United Kingdom
H2020 ICT projects named DANCE and WHOLODANCE in whichthe authors of this paper are participating, a custom EyesWeb-basedso�ware application has been developed. �is application pro-cesses the raw IMU sensor data and extracts low and higher levelmovement features. �ese features are sent via the open sound com-munication protocol (OSC) to the composition and piano controlso�ware. �is la�er so�ware has been developed in the Supercol-lider programming environment. A third piece of so�ware is acustom developed swarm simulation program. �is program hasalso been speci�cally developed for the piece and serves mediatessemi-autonomous system between movement qualities and pianogestures. Figure 2 depicts a schematic representation of this techni-cal setup.
Figure 2: Schematic Depiction of Technical Integration ofthe Sensing, Communication, Computation and Piano Con-trol Systems.
3.3 Soni�cation of Movement�alitiesAs mentioned previously, the realisation of Piano&Dancer wasstrongly motivated by the desire to address the problem of howmovement qualities can be related to the morphology and articula-tion of sounds in such a way that the dancer’s movement can beinferred from the resulting sounds.
Each of the composed scenes in Piano&Dancer develops a par-ticular movement quality. �ese movement qualities are alwayslinked to a particular musical material or algorithmic approachto composition. Moreover, the choreographic structure highlightsthese relationships. For example by focusing on certain body partsthat are associated to the actuation of a group of piano keys. Adetailed description of the choreographic and musical aspects ofthe piece may be found in [6].
Of particular importance in the piece are the movement qualitiesEnergy, Weight, Smoothness, and Dynamic Symmetry of Smooth-ness. �ese qualities can be easily associated to particular embodiedmetaphors or image schemas. For instance Energy is tightly con-nected to embodied metaphors of force and constitutes a transversalaspect of movement. Energy also forms a key aspect of any hu-man gesture that gives rise to sound morphologies. Energy canbe combined with other qualities in order to obtain a movementanalysis that is more in agreement with an intuitive appreciation
of a dancer’s gestural actions. �e Weight quality is based onLaban’s weight e�ort and describes how gravity in�uences a move-ment. Accordingly, Weight is directly connected with the verticalityimage schema that has been previously mentioned. Two of theaforementioned qualities, Smoothness and Dynamic Symmetry ofSmoothness, as de�ned within the framework of the EU H2020 ICTprojects DANCE and WHOLODANCE [9], will now be described inmore detail in combination with the soni�cation and choreographicstrategies that have been developed to exploit them.
3.3.1 Smoothness. Smoothness corresponds to a concept frombiomechanics and is de�ned as minimum jerk [9]. �e movementof a joint is considered smooth when no abrupt changes in accelera-tion occur. When taking into consideration the activity of multiplejoints, smooth movement would correspond to a coordinated wave-like propagation pa�ern through several body joints [26]. �e na-ture of this quality is closely related with the notions of continuity,�uidity and predictability in case of high levels of Smoothness andconversely to discontinuity, shakiness, unpredictability, accident orsurprise for low levels of Smoothness. In Piano&Dancer, the analysisof Smoothness is performed on the acceleration data obtained fromthe dancer’s wrist movements. �e dancer develops choreographicmaterial to alternate between high and low levels of Smoothness.�ese levels are linked to the organisation of musical pitch. Bydoing so, a gamut of contrasting musical entities can be created,ranging from smooth glissandi to jumps across distant degreeswhich split or break the continuity of music lines. �e contrastingquality of the resulting music can be clearly seen in the real timescore transcription of the corresponding scene in the performance(see Figure 3). Smoothness is mapped to a sequence of evolvingharmonic �elds. �e �rst �eld is purely chromatic to clearly exposethe relationship between the quality and the response of the pianokeys. Very smooth movements only set into motion adjacent keys.�is chromatic �eld evolves into a non-octavating scale and itsinversion, a sieve in the Xenakian conception that has been spe-cially devised for the piece. In an non-octavating scale, notes do notrepeat a�er covering the span of an octave but they maintain inter-vallic consistency. �is o�ers the interesting possibility of not onlyassigning di�erent sections of the scale to the movement of di�er-ent joints of the dancer’s body but also to create echoing sonoritiesas the dancer creates slides or jumps across the octaves of the key-board. Continuous smooth movements are not only re�ected in thepitch but also also amplitude domain as accelerando-deccelerandoor crescendo-decrecendo. In the corresponding scene, the dancerstands behind and above the piano (see Figure4). �is setup putsinto focus the strong connection between the dancer’s bodily move-ments and the movements of the piano keys. �e movement ofthe keys visually convey the Smoothness quality in that keys areactuated only locally when the dancer’s movements are smoothwhereas jerky movements trigger the depression of multiple keysthat spread over the entire keyboard.
3.3.2 Dynamic Symmetry. Dynamic Symmetry is a higher levelquality that di�ers from static or postural symmetry in that itincludes dynamic and temporal aspects. Dynamic Symmetry isbased on an analysis of the degree of coordination and dynamicsof multiple moving body parts [8]. �e skill necessary to maintainhigh or low levels of Dynamic Symmetry is important not only in
MOCO’17, 28-30 June, 2017, London, United Kingdom P. Palacio and D. Bisig
Figure 3: Contrasting Score Sections Corresponding to High(top image) and Low (bottom image) Movement Smoothness
Figure 4: Dancer Standing Behind the Piano. Her move-ment material alternates between low level and high levelof Smoothness.
contemporary dance but also in classical ballet. In Piano&Dancer,Dynamic Symmetry is derived from a comparison of the level ofSmoothness between the acceleration of pairs of joints. �e pianosoni�cation mirrors in the auditory domain the degree of DynamicSymmetry in the dancer’s movements. �e piano soni�cation modeldeveloped for this quality is based on the equivalence of symmetryand periodicity, where symmetry corresponds to repetitions inspace and periodicity to repetitions in time. �e model exploresmultiple possible states from periodicity to aperiodicity in order tomirror the degree of Dynamic Symmetry in the dancer’s movements.�ese contrasting states can easily be distinguished in a real timescore transcription (see Figure5). �e degree of Dynamic Symmetrybetween body parts is re�ected musically not only in the resultingpitch and amplitude dynamics but also in the inner structure of theevolving musical modes that shape the harmonic content and colorof the music. �ese structures correspond to a selection of modes oflimited transposition [22] that are composed of symmetric modesin which the last note of each group is always the �rst one of thenext group. �e degree of symmetry of a mode is determined by thenumber of transpositions it allows before it arrives at its startingposition. For instance mode 1 (the famous whole tone scale) hasonly two transpositions, whereas mode 6 has six transpositions.In the corresponding scene of Piano&Dancer, the level of DynamicSymmetry is mapped to the degree of symmetry of a mode, ranging
from mode 2 (three transpositions) to mode 6 (six transpositions).Accordingly, the dancer can modulate from one mode to anotherbased on the similarity of the smoothness of her wrist movements.
Figure 5: Contrasting Score Sections Corresponding to Low(top image) and High (bottom image) Movement DynamicSymmetry
By providing an interaction scenario that mediates between thekinaesthetic and visual domain of body movement on one hand andthe acoustic domain of algorithmic composition and piano playbackon the other hand, the dancer becomes able to transfer articulatedgestural expressions into music without the necessity for her toconsciously pay a�ention to and plan the musical consequences ofher movements. Rather, it is up to the music composer to specify thecompositional and sonic principles of the music, the automation ofwhich is incorporated and �nally delegated to the algorithmic andmechanical mechanisms of the computer and piano, respectively.As a result, the dancer can focus on the creative and aestheticprinciples that lie fully within her own area of expertise whilerelying on the compositional expertise embedded in the interactivemusical system to respond in a musically meaningful way to herown performance.
�e incorporation of algorithmic and generative methods intothe mediating layers between movement and music allows thechoreographic and compositional elements of the performance topreserve their intrinsic aesthetic principles while at the same timeremain connected through a strong causal relationship. �ese algo-rithmic abstractions are in�uenced or perturbed in di�erent waysby the bodily movements of the dancer. �e design of these algo-rithmic abstractions has been carried out in such a way that thequalities of the dancer’s movements are transferred into composi-tional structures and sonic morphologies that convey aestheticallyand metaphorically similarly qualities in the acoustic domain. Amore complete description of these mediating algorithmic layersand musical abstractions can be found in [6].
Piano&Dancer - Interaction Between a Dancer and an Acoustic Instrument MOCO’17, 28-30 June, 2017, London, United Kingdom
Some of these algorithms explore the properties of �nite groups.�is for instance is the case for the automated creation of invertedtranspositions on the same fundamental note. As an example, theapplication of automated transposed inversions depends on therotation of the dancer’s ankles and wrists. For each joint rotation,the algorithm returns a new version of an array of numbers corre-sponding to an inverted transposition with the same fundamentalnote. Based on the direction of a wrist’s rotation, the resultingchord is either in a closed or open position.
Other approaches make use of probability distributions such asBeta or Exponential distributions. �ese distributions are used forexample for the creation of random walks, which are controlledby the level of Smoothness [9] of the dancer’s current movements.In this case, the step size of the random walks is proportional tothe jerkiness of the dancer’s movement. Furthermore, the direc-tion of the random walks depends on the direction of the angularacceleration of the movement. �e set of probability distributionsare employed to shape the density of events and also to distributethem across di�erent harmonic �elds. Each distribution has its ownspeci�c characteristics and can be associated with di�erent musicalaspects. For example, Cauchy and Gaussian distributions are com-bined and superposed and their means are assigned to harmonicnodes or tonal centers that are associated to speci�c body joints.�e levels of movement qualities of a particular joint may be usedto perturb the spread or deviation from the mean that de�nes thesedistributions. �is causes the musical output to oscillate betweenharmonic and more dissonant states. A similar application may bedeveloped in the temporal domain to organise durations that movebetween predictable and unpredictable or random rhythms. Ofmusical interest are also Beta distributions since they can be usedto generate values that cluster at the lower and upper boundary ofthe distribution range. �e mapping of these values on the pitchof notes causes a frequent appearance of highest and lowest notes.�ese notes are of critical importance in the way music is perceived.�e dancer’s movement qualities can be used to control the shapeof the beta probability distribution and thereby a�ect the frequencyof occurrence of extreme note pitches.
A more sophisticated algorithmic layer is provided by a swarmsimulation that has been implemented using the ISO programminglibrary [5]. By integrating swarm simulations as intermediarylevel between movement analysis and piano control, the interactivesystems gains the capability to create complex musical material ina self-organised manner. In this setup, the dancer is no longer ableto directly and fully control the behaviour of the algorithmic layerbut rather assumes the role of an improvisation partner with anarti�cial and autonomous musical agency.
�e repertory of swarm behaviours that are provided by theprogramming library has been expanded with several additionalbehaviours. �ese behaviours have been developed to meet thespeci�c requirements to provide control data for the creation ofdiscrete and note based musical forms (see Figure6).
�e most important of these behaviours are: A discretisation be-haviour that maps any continuous agent parameter such as positionor velocity to a prede�ned set of discretised values. A cohesion be-haviour that permits the speci�cation of axis aligned o�sets amongthe positions of neighbouring agents. �is e�ect permits the real-ising of chord-like groupings within a swarm. A neighbourhood
Figure 6: Custom Agent Behaviours for Swarm Simulations.From le� to right: discretisation behaviour, axis aligned o�-set behaviour, sequencing behaviour.
behaviour that encodes the positions of neighbouring agents inspherical coordinates in order to simplify the distinction betweendistance and orientation relationships within a swarm. �e distancecan be used for instance to control the intervallic relationships be-tween notes whereas the orientation can control the permutationsof chords. A sequencing behaviour that triggers a timed series ofmodi�cations to a particular agent parameter. �e purpose of thesequencing behaviour is to generate control data that exhibits amotivic form.
An example of how the swarm simulation mediates betweenbodily activity and piano control is provided by a scene in whichthe choreography is restricted to the hands which oscillate be-tween high and low degrees of Dynamic Symmetry. In this scene,the Dynamic Symmetry of Smoothness between both wrists of thedancer controls the cohesion of a swarm whose agents’ positionsare mapped to piano keys. Due to the fact that the swarm gathersinto oscillating clusters when the cohesion among agents is highwhereas it spreads through outwards trajectories when cohesion islow, its mapping onto piano keys causes the piano to mirror bothvisually and acoustically the dancer’s level of Dynamic Symmetry.
All the algorithms that have been described so far map theirnumerical output onto one of several prede�ned harmonic modalsystems before being translated into pitch values. �ese modalsystems shape part of the aesthetic character of the work. A moredetailed description of these musical structures may be found in[6].
4 DISCUSSION AND CONCLUSIONMusic performance is a complex and multidimensional phenomenathat involves not only a capability for sophisticated conceptual-isation and formal structuring on the part of the composer butalso highly skilled body awareness, movement co-ordination, andaesthetic sensitivity on the part of the instrument player. Our cho-sen approach of creating an interactive gesture-based relationshipbetween dancer and piano serves to expand the creative and ex-pressive freedom of movement on the side of the performer whilemaintaining a clear correspondence between movement and ex-pression in the kinaesthetic, visual and acoustic domain.
By connecting high level movement qualities that play an impor-tant role in the dancer’s body awareness to the generation of sonicmaterial, the acoustic feedback provided by the interactive systemmay help to emphasise and foreground otherwise hidden choreo-graphic aspects for both the dancer and the audience. �is is madepossible mainly because of two approaches: the abolishment of a
MOCO’17, 28-30 June, 2017, London, United Kingdom P. Palacio and D. Bisig
direct touch-based interaction with the piano and the extraction ofhigher level qualities from the dancer’s movements. We considerthese two approaches to constitute a minimum requirement forallowing choreography to free itself from the constraints of normalmusical playing gestures while at the same time to maintain a clearcorrelation between physical gesture and musical gesture.
Furthermore, we discussed how image schemas that form thebasis for our higher level cognitive capabilities can help to grounddesign principles for interactive musical systems in the bodily fa-miliarity of its users. In the context of dance, such an approachenables the dancer to fully rely on her movement expertise whilemoving through a sensing and sounding interactive environment.
�e paper has also brie�y introduced a combination of algorith-mic and swarm-based generative techniques for creating note-basedmusical structures. �is algorithmic layer establishes a networkof direct and indirect cause and e�ect relationships that connectthe dancer’s movements to a wide spectrum of di�erent sonic re-sults. �e usage of such an algorithmic layer is a�ractive since itprovides both precise compositional control over the musical resultwhile also o�ering the possibility for the emergence of complexand surprising musical pa�erns.
Finally, we would like to mention that the realisation of Pi-ano&Dancer constitutes for us a bold step away from our previousartistic activities. Beforehand, we had always employed withinour collaboration exclusively computer-based synthetic musicalinstruments for creating the interactive musical aspects of the per-formance. One goal of this publication was to clarify both to thereader but also for ourselves the artistic motivation and consider-ations that led to our undertaking of this new creative adventure.One of the most exciting aspects in the creation of this new piececoncerns the design of an interactive sensing environment in whichthe physical presence and gestural qualities of the dancer and theacoustic instrument complement and enrich each other in both astraight forward and striking manner. Furthermore, the richnessof the natural acoustic sounds, the residual noises caused by themovements of the piano keys, the large and nuanced movementsof the dancer, and the visual relationship and friction between amechanical instrument and a human dancer create a stage situationthat is densely populated by audiovisual events without ever be-coming overwhelming. �ese observations provided an additionalmotivation and increased engagement for realising Piano&Danceras compared to the realisation of more strongly computer-basedinteractive media contexts.
REFERENCES[1] Rafael Perez Arroyo. 2004. Music in the age of the pyramids: La musica en la era
de las piramides. Natural Acoustic Recordings.[2] Enda Bates. 2009. �e Composition and Performance of Spatial Music. Ph.D.
Dissertation. Trinity College Dublin.[3] Gregory Bateson. 1972. Steps to an ecology of mind: Collected essays in anthropol-
ogy, psychiatry, evolution, and epistemology. University of Chicago Press.[4] Andreas Bergsland and Robert Wechsler. 2015. Composing interactive dance
pieces for the motioncomposer, a device for persons with disabilities. In Pro-ceedings of the international conference on New Interfaces for Musical Expression.20–23.
[5] Daniel Bisig, Martin Neukom, and John Flury. 2008. Interactive Swarm Orchestraa Generic Programming Environment for Swarm Based Computer Music.. InICMC.
[6] Daniel Bisig, Pablo Palacio, and Muriel Romero. 2016. Piano&Dancer. In Proceed-ings of the 19th Generative Art Conference. 138–154.
[7] Alethea Blackler, Vesna Popovic, and Doug Mahar. 2010. Investigating users�intuitive interaction with complex artefacts. Applied ergonomics 41, 1 (2010),72–92.
[8] Antonio Camurri, Corrado Canepa, Nicola Ferrari, Maurizio Mancini, RadoslawNiewiadomski, Stefano Piana, Gualtiero Volpe, Jean-Marc Matos, Pablo Palacio,and Muriel Romero. 2016. A system to support the learning of movement qualitiesin dance: a case study on dynamic symmetry. In Proceedings of the 2016 ACMInternational Joint Conference on Pervasive and Ubiquitous Computing: Adjunct.ACM, 973–976.
[9] Antonio Camurri, Gualtiero Volpe, Stefano Piana, Maurizio Mancini, RadoslawNiewiadomski, Nicola Ferrari, and Corrado Canepa. 2016. �e dancer in the eye:towards a multi-layered computational framework of qualities in movement.In Proceedings of the 3rd International Symposium on Movement and Computing.ACM, 6.
[10] William CD-ROM FORSYTHE, Nick HAFFER, and Volker KUCHELMEISTER.Improvisation technologies: a tool for the analytical dance eye. Musica �omWillems, Maxin Franke; direcao de fotogra�a �omas Lovell Balogh, Jess Hall.SI: RD-Studio, 1999. 1 CD-ROM multimıdia. (��).
[11] M Coniglio and D Stoppielo. 2007. MidiDancer: A Wireless Movement SensingSystem. Available on-line at www. troikaranch. org/mididancer. html. AccessedJanuary (2007).
[12] Brent Gillespie. 1999. Music, Cognition, and Computerized Sound: An Introduc-tion to Psychoacoustics, chapter 18, Haptics. (1999).
[13] Richard Graham and Brian Bridges. 2015. Managing musical complexity withembodied metaphors. In Proceedings of the International Conference on NewInterfaces for Musical Expression (NIME). Louisiana State University.
[14] Jorn Hurtienne and Johann Habakuk Israel. 2007. Image schemas and theirmetaphorical extensions: intuitive pa�erns for tangible interaction. In Proceed-ings of the 1st international conference on Tangible and embedded interaction. ACM,127–134.
[15] Ksenia Kolykhalova, Paolo Alborno, Antonio Camurri, and Gualtiero Volpe. 2016.A serious games platform for validating soni�cation of human full-body move-ment qualities. In Proceedings of the 3rd International Symposium on Movementand Computing. ACM, 39.
[16] George Lako�. 1990. �e Invariance Hypothesis: is abstract reason based onimage-schemas? Cognitive Linguistics (includes Cognitive Linguistic Bibliography)1, 1 (1990), 39–74.
[17] Marc Leman. 2012. Music and schema theory: Cognitive foundations of systematicmusicology. Vol. 31. Springer Science & Business Media.
[18] Aaron Levisohn and �ecla Schiphorst. 2011. Embodied engagement: Supportingmovement awareness in ubiquitous computing systems. Ubiquitous Learning:An International Journal 3 (2011), 97–111.
[19] Mary Michelle Mainsbridge. 2016. Body as instrument: an exploration of gesturalinterface design. Ph.D. Dissertation.
[20] Erin Manning. 2006. Prosthetics making sense: dancing the technogenetic body.the Fibreculture Journal 9 (2006).
[21] Olivier Messiaen. 2002. Traite de rythme, de couleur, et d’ornithologie:(1949-1992):en sept tomes. Alphonse Leduc, Paris.
[22] Olivier Messian. 1994. �e technique of my musical language. Alphonse Leduc,Paris.
[23] Jean Newlove and John Dalby. 2004. Laban for all. Taylor & Francis US.[24] Kia Ng. 2012. mConduct: transcending domains and distributed performance. Ann
Arbor, MI: Michigan Publishing, University of Michigan Library.[25] Ricardo Pedrosa and Karon MacLean. 2008. Perceptually informed roles for
haptic feedback in expressive music controllers. In International Workshop onHaptic and Audio Interaction Design. Springer, 21–29.
[26] Stefano Piana, Paolo Alborno, Radoslaw Niewiadomski, Maurizio Mancini,Gualtiero Volpe, and Antonio Camurri. 2016. Movement �uidity analysis basedon performance and perception. In Proceedings of the 2016 CHI Conference Ex-tended Abstracts on Human Factors in Computing Systems. ACM, 1629–1636.
[27] Joseph Rovan and Vincent Hayward. 2000. Typology of tactile sounds and theirsynthesis in gesture-driven computer music performance. Trends in gesturalcontrol of music (2000), 297–320.
[28] Jan C Schacher and Daniel Bisig. 2014. Watch �is! Expressive Movement inElectronic Music Performance. In Proceedings of the 2014 International Workshopon Movement and Computing. ACM, 106.
[30] Balint Andras Varga. 1996. Conversations with Iannis Xenakis. Faber and Faber,London.
[31] Elizabeth Waterhouse. 2007. Technological Artifacts: �e Transmission ofWilliam Forsythe�s Choreographic Knowledge. In Congress on Research in Dance(CORD) 40th Annual Conference, Choreographies of Migration: Pa�erns of GlobalMobility.
[32] Trevor Wishart and Simon Emmerson. 1996. On sonic art. Vol. 12. PsychologyPress.
![Page 1: Piano&Dancer - Interaction Between a Dancer and an ... · as paralanguage and give rise to expressive aspects of speech that seem to be independent of verbal content [3]. In a similar](https://reader043.documents.pub/reader043/viewer/2022041010/5eb886a4eb00c40c99371dd9/html5/page/1.jpg)
![Page 2: Piano&Dancer - Interaction Between a Dancer and an ... · as paralanguage and give rise to expressive aspects of speech that seem to be independent of verbal content [3]. In a similar](https://reader043.documents.pub/reader043/viewer/2022041010/5eb886a4eb00c40c99371dd9/html5/page/2.jpg)
![Page 3: Piano&Dancer - Interaction Between a Dancer and an ... · as paralanguage and give rise to expressive aspects of speech that seem to be independent of verbal content [3]. In a similar](https://reader043.documents.pub/reader043/viewer/2022041010/5eb886a4eb00c40c99371dd9/html5/page/3.jpg)
![Page 4: Piano&Dancer - Interaction Between a Dancer and an ... · as paralanguage and give rise to expressive aspects of speech that seem to be independent of verbal content [3]. In a similar](https://reader043.documents.pub/reader043/viewer/2022041010/5eb886a4eb00c40c99371dd9/html5/page/4.jpg)
![Page 5: Piano&Dancer - Interaction Between a Dancer and an ... · as paralanguage and give rise to expressive aspects of speech that seem to be independent of verbal content [3]. In a similar](https://reader043.documents.pub/reader043/viewer/2022041010/5eb886a4eb00c40c99371dd9/html5/page/5.jpg)
![Page 6: Piano&Dancer - Interaction Between a Dancer and an ... · as paralanguage and give rise to expressive aspects of speech that seem to be independent of verbal content [3]. In a similar](https://reader043.documents.pub/reader043/viewer/2022041010/5eb886a4eb00c40c99371dd9/html5/page/6.jpg)
![Page 7: Piano&Dancer - Interaction Between a Dancer and an ... · as paralanguage and give rise to expressive aspects of speech that seem to be independent of verbal content [3]. In a similar](https://reader043.documents.pub/reader043/viewer/2022041010/5eb886a4eb00c40c99371dd9/html5/page/7.jpg)
![Page 8: Piano&Dancer - Interaction Between a Dancer and an ... · as paralanguage and give rise to expressive aspects of speech that seem to be independent of verbal content [3]. In a similar](https://reader043.documents.pub/reader043/viewer/2022041010/5eb886a4eb00c40c99371dd9/html5/page/8.jpg)
