Maura McDonnell

Visual music – a composition of the ‘things themselves’.

Music and Media Technologies, Trinity College Dublin http://www.mee.tcd.ie/mmt

mmcd@soundingvisual.com http://www.soundingvisual.com

Introduction

Putting together the worlds of sound and image as a unified field of video projection and sound is a growing area of contemporary practice for filmmakers, video artists, animators, music composers and music technology artists. It is an area of activity that comes under the broad area of sonic arts. Within this broad area of practice, however, there is emerging a distinct type of video projection and sound that presents a visual composition of image elements that do not reference anything other than the things themselves of the image and a music composition that also does not reference anything other than the things themselves of music. In such compositions, both visual and music things explore an interaction and interplay between these visual and musical things themselves. The term ‘visual music’ could be used to describe more accurately these works, precisely because it is the interplay of visual things and music things that is the vital consideration in the composition and crafting of the work. This paper will focus on elaborating some of these things themselves of the visual composition in the context of contemporary practice. The discussion of the music things and a discussion of the interplay of visual and music things will be examined in a future paper. However, by focusing on the visual things and identifying examples from contemporary practice, this will elaborate further the activity of visual music and assist in ensuring visual music works are considered both as a valuable and an exciting area of the sonic arts.

Visual Music Term

Contemporary use of the term ‘visual music’ has arisen mainly from the presentation of recent exhibitions and events, which present contemporary visual music works, and in some cases document these exhibitions with a catalogue or book (Brougher et al., 2005; Ox et al., 2006; McDonnell, 2007; Rainer et al. 2009). Individual artists, academics and composers, who have been working closely with sound and image material in their own practice, have been doing so for quite some time, and indeed for quite some time before some of these exhibitions have taken place. An earlier discussion about the definition of the term arose on the Iotacenter Discussion Group list (Ross, 1999) amongst practioners in the audio visual field. Earlier again in 1986, William Morritz (1986, 2004), uses the term visual music in his research and writings, and documented the start of a history for the area of visual music where he recognised — in the area of historical absolute film or abstract cinema, and in particular in his documentation and analysis of the filmmaker

Oskar Fischinger — that there was a particular aesthetic of visual music arising in these works that provides the beginnings of principles of aesthetics for new artists setting out to compose visual music. Roger Fry first used the term in 1912 to describe the particular ‘closely knit structure’, the colour oppositions of the abstract paintings of Wassily Kindinsky (Spalding, 1980, p168). Some contemporary artists/ filmmakers /composers, who would describe their main body of audio-visual work and approach with different terms, for example, animation cinema (Hébert, 2010), audiovisual animation, audiovisual composition, cross-media, electronic cinema, electroacoustic movies (Cope), live multimedia, intermedia performance, intermedia composition (Kapuscinski et al. 2009), object animation and motion painting (Maxwell), video music (Piché), audio-visual artworks (Battey), visualisation of music as an abstract phenomenon (Ox), would consider some of their practice and works to be visual music works, and to be described as such, even if they do not deploy the term ‘visual music’ to describe all of their works.

Defining Visual Music – Identifying Visual Music Practice

Recent visual music exhibitions and symposiums indicate, however, that the practice of visual music is growing, for instance, the Visual Music Marathons (Miller, 2007) held in Boston, 2007 and New York 2009, the Punto y Raya Festivals held in Madrid 2007 and Barcelona, 2009, the See This Sound exhibition (See This Sound, 2009) held in Austria and the Seeing Sound: Practice-led Visual Music Research Symposium (Hyde, 2009) held in the UK and many others. As the practice of visual music is growing, so too is research and writing on the field (Computer Music Journal, 2005; Daniels et al., 2009; Lund et al., 2008). It is hard, nevertheless, to pin down and to agree precisely what visual music is, as some practioners question the term and others debate its definition. This is due to a number of factors, such as, for instance: what technology is being employed to create the work; what background the artists/composers/practioners come from, whether it is (primarily) a music composition, filmmaking, vj-ing, visual art, computer engineering, computer programming background; what method is the visual music work being presented to audiences, for example, is it being presented as a fixed media video and sound screening, is it a performance of musicians and video, is it an installation, is it a live improvisation of visuals and music, is it a real-time interactive rendering of images and sound. Some practioners also focus on the sensory aspect of combining visuals and audio. Here, the focus of the work is on its final appeal to the senses. This, in a way, adds to some confusion in relation to defining visual music. All these discussions, however diverse they may be, do help, nonetheless, to define and to understand a phenomenon of art and music activity that is actually being created and taking place and that is an active area of practice. And as more artists and composers who create ‘visual music’ works document their approaches and thoughts on the area, the more this will contribute to the developing area of visual music practice. One example of such formalised research is taking place at the CCRMA/Intermedia Performance Lab at Stanford University led by Jaroslaw Kapuscinski (2001), where detailed studies investigate the intermedia areas of exploring the relationship between visual art and music (figure 1). For definitions of visual music see Ox et al. (2006), McDonnell (2007), Lund et al. (2008). For useful historical discussions of precursors in the field see Elder (2008), Keefer (2005), Fred Collopy’s website (http://rhythmiclight.com/). The organisations, the iotaCenter (iotaCenter) and the Center for Visual Music (Center for Visual Music), based in the US promote both historical and contemporary practice in the

visual music field and they are forums for the gathering of resources, curating exhibitions and events, promoting contemporary practice, preserving historical films, and building a community of all those interested in the field of visual music and ‘experimentation, abstraction, and animation’ (iotaCenter).

Figure 1: Jaroslaw Kapuscinski & Javier Sanchez, Counterlines (2009)

Jack Ox and Cindy Keefer, then, provide a useful, broad definition of contemporary visual music, defining visual music practice as being characterised into four differently formed visual structures. This definition arose from their curating of New York Digial Salon’s Abstract Visual Music Project. In summary, they distinguish one visual structure to be, “a visualization of music which is the translation of a specific musical composition (or sound) into a visual language”; the second visual structure is a “time based narrative visual structure that is similar to the structure of a kind or style of music”; the third visual structure is defined as a “direct translation of image to sound or music”; and the fourth and final visual structure defined is a “visual composition that is not done in a linear, time-based manner, but rather something more static like a 7’ x 8’ canvas”(Ox et al. 2005). This is a good starting point for any enquiry into visual music.

Visual Energy

For the remainder of this paper, the discussion in relation to the things themselves in a visual music work is going to concentrate on the ‘visual things’. The visuals in the final product — regardless of how and in what way they were composed, that is to say, whether they were they composed before the music or after the music or alongside — seem to be directed towards the music, music is the impetus and the ground that drives the energy of the visual. The main energy of a visual music piece is this visual energy. The visual energy of a visual music piece bursts out of the screening frame, as an audience we are no longer bound to a chronological rendition of frame images. The visual energy is, the things themselves in the image, these things have their own trajectory of appearance and disappearance, their own dynamics, their own rhythm, of being directed into the depth of the frame, or of having their own life span and momentum.

There can be multiple things made visible to us, and each thing contributes to the overall visual effect as they change and evolve over time, interact with each other, interplay with the elements of music in the sound track. As in absolute film, the imagery is objective. The visuals are objects; they are things that reference themselves – the things themselves of the image.

Visual Things

What, exactly, are these visual things with which one composes in a visual music work? The visual things can be simple elements of a picture drawing and graphic design, or they can be more complex figures crafted from the simple elements of picture and graphic. They can also be more painterly, focusing on filling the screen space with a multitude of colours and textures. Computer images allow us to have a precise control of computer image parameters, these parameters become new visual things accessible by the artist and results in images that can consist in precise patterns and in ordered shapes to images consisting of a multitude of dots and lines, forever changing and transforming across time.

Examples of Pictorial Elements

Figure 2: Javier Sanchez, Unorthogonality (2009)

Simple pictorial, drawing and graphic elements consist of: straight lines; curved lines; dots; simple shapes; use of colour, drawing marks of all shapes, lengths, and sizes; scribble and scratch marks; pictorial figures; geometric figures, flat and gradient blocks of colour. Historical works used a range of mechanical devices and techniques to record pictorial and graphic elements for their films, for example, Oskar Fischinger’s Studies films (Fischinger, 1929-1932), Hans Richter’s Prelude (Richter, 1919), Kurt Schwerdfteger’s Reflektorische Farblichtspiele (Schwerdfteger, 1922). More modern works, use different technological means to realise their visuals, for example by drawing directly on film (Lye, McLaren), or the video and early computer graphics experiments

for generating dots of light and graphical blocks and building these into geometric shapes and patterns (Whitney, Cuba).

Figure 3: Oerd van Cuijlenborg, Jazzimation (1999). Music: Jeroen van Vliet.

Figure 4: Chiaki Watanabe, 1/3 (one over three) vol.1. (2006). Sound: Tristan Perich and Sylvia Mincewicz. © Chiaki Watanabe.

Contemporary visual music works visualise similar pictorial image elements, and have access to both traditional techniques of image and film-making, as well as having access to a whole plethora of digital technologies to realise their image elements. Javier Sanchez’s video Unorthogonality (2009) uses simple lines to explore spatial and musical movement (figure 2). Oerd van Cuijlenborg’s Jazzimation film (1999), in a style similar to historical works of Len Lye and Norman McLaren, uses direct film animation techniques creating lines, squiggles and scribbles to create his visual to music composition, aligning lines to music instruments and music elements (figure 3). Chiaki Watanabe’s 1/3 (one over three) vol.1 (2006) in her computer images for live performance, experiments with one-bit as an art expression based on one-bit technology, the emphasis is on using a single bit of information such as one-bit color, one-bit code and a one-bit note (Watanabe 2006). Neil O’Connor’s video The Glistening Bride (O’Connor 2004) uses simple curved digitally created graphic blocks of colour to change the scale parameter of the blocks over time, in accordance with the dynamics and intonation of the music instruments in the piece

(figure 5). Jane Cassidy’s visual music work, The night after I kicked it (Cassidy, 2008) uses a series of lines to create a minimalised spatialised image composition that interacts with the spatialised sounds in the music composition (figure 6).

Figure 5: Neil O'Connor, The Glistening Bridge (2004). Music: Neil O'Connor

Figure 6: Jane Cassidy, The Night After I Kicked It (2008). Music: Jane Cassidy

Examples of Complex Figures

More Complex visual figures and forms can be created with these pictorial and graphical building blocks, and these pictorial forms become new visual objects or figures, that we come to know and recognise as they are transformed over time. The film Symphonie Diagnonale by Viking Eggeling’s (Eggeling, 1924) is an example. Eggeling creates visual forms from basic visual elements and using the idea of musical counterpoint elaborates their arrangement in time. He calls these forms, “themes” or ‘instruments”. Larry Cuba’s computer animated film Calculated Movements used computer programming to create solid areas and shape volumes (figure 7). The film consists of a “choreographed sequence of graphic events constructed from simple elements repeated and combined in a hierarchical structure”; this work has a precise mathematical structure in the design of the graphic events and their unfolding in time. (Cuba, 1985)

Figure 7: Larry Cuba, Calculated Movements (1985). Sound: Larry Simon, Craig Harris, Rand Weatherwax

Pierre Hébert’s recent film Triptych (2009), was made from a series of live animation performances, using lines, he creates three simple figurative forms positioned in their triptych positions that become more complex over the temporal evolution of the piece (figure 8 and 9). These figurative forms consist of several lines that are part of the visual form and are independent from it, by having their own action, the image elements coalesce to form figures, that then transform and reveal motions and actions that are aligned to the drama of the music.

Figure 8 and 9: Pierre Hébert, Triptych (2009). Music: Bob Ostertag

LIA’s video Construction 76 (LIA, 2008) builds beautiful shapes and patterns with lines and simple graphic elements, and the sounds are synchronised with these graphic elements (figure 10). The hand-painted experimental abstract animations of Stephanie Maxwell, such as in her film All That Remains, (figure 11) comprises “an intricate mosaic of sequences of animated abstract images and musical passages that create a chaotic yet coherent and tightly choreographed portrayal of figurative matter in perceptual decomposition” (Maxwell, 2006).

Figure 10: LIA, Construction 76 (2008). sound: @c

Figure 11: Stephanie Maxwell, All That Remains (2006). Music: Michaela Eremiasova

Examples of Painterly Approaches

A more painterly fine art approach can also be taken to creating the elements of image. The whole screen space becomes a ground and surface for full screen activity, for example, filled with washes and textures of colours, merged forms, a multitude of lines, a sparsity or density of image particles, that change, merge, coalesce, collide, interact like a large abstract painting canvas that has come to life over time and with the addition of music. The final result is one of a whole visualscape of imagery, a visual environment, and a visual odyssey, in some instance, made incredibly powerful with the addition of time. Digital processing techniques have allowed us to bend and mould imagery in the computer to become the basis of the most complex visualscapes and create both 2D and 3D visual environments. Jean Detheux’s develops his digital abstract expressionism to create incredible imagery as it unfolds in time and interconnects in a multiple of ways to the music. The energy in his works are so tightly aligned to the music it follows, they are the most incredible examples of visual music. His latest work Visual Abstract explores this energy with a multitude of changes and variations, which is similar to the multitude of changes and variations in the timbre of the music instrument or the melody or rhythmic pattern or the dramatic dynamics in the music (figure 12)

Figure 12: Jean Detheux, Visual Abstract (2010). Music: Pierre Jalbert

Examples of Computer image with electroacoustic music composition

Technological advances have let us access micro parameters of image, through a wide range of techniques from image processing and image manipulation, to generative images that are created from code and programming or from a mapping of data, to even audio data generating images. Such levels of parametric access provides us with the means to create image elements and manipulate these image elements into whatever form we or the software or programme can assist us to do so. The following examples are works

where a composer composes both the images and music. Diego Garro’s video Patah (Garro, 2009) creates image particles to match the granular grains in his music composition; similar processes of transformation and speed are applied to both images and sound. His music composition is an electro-acoustic composition, and his image composition has a similar feel, an electro visualscape (figure 13).

Figure 13: Diego Garro, Patah (2009). Music: Diego Garro

Dennis Miller also creates complex visual forms as well as composing the electro acoustic music composition. His latest work Echoing Spaces, “explores a number of virtual environments in which echoing, repeating lines and forms emerge (figure 14). The visual imagery employs a number of similarly shaped elements that appear in overlapping, morphing configurations.” (Miller, 2009). Bret Battey’s Sinus Aestrum computer realised video (figure 15) is an incredible unfolding of “12,000 individual points, which are continually transformed and warped, restrained and released, without cuts, to form compound, multi-dimensional waves of activity” (Battey, 2009).

Figure 14: Dennis H. Miller, Echoing Spaces (2009). Music: Dennis H. Miller

Figure 15: Bret Battey, Sinus Aestrum (2009). Sound: Bret Battey

Examples of Real things as Visual Objects

Figure 16: Jaroslaw Kapuscinski, Juicy (2009). Collaboration: John Edmark

The things of image refer, in many cases, to these more abstract marks and basic units of drawing and picture making and painterly effects, but the things of image can be any visual object defined by the artist, for example, real world photography, or moving image elements, or montages of images can be used as a thing in the visual composition, but the photograph or piece of live action footage is not being used to reference the content

of the image, rather they become the building blocks of a composition that references the composition of image elements in time. For example, Jaroslaw Kapuscinski’s Juicy (2009) and Huckleberry Lain’s Parallel (Lain, 2010). Kapuscinki’s Juicy is a Performance/installation for Disklavier piano and computer controlled image and sound, in collaboration with John Edmark (figure 16). It uses images of fruit pieces as the image object. With a close alignment to the music and inter-media interplay of images and sound, “sounds and images are given equal importance and are developed either simultaneously or in constant awareness of each other” (Kapuscinski, 2001). Huckleberry Lain’s Parallel (Lain, 2010) also uses real imagery and units of moving image in this work, mixed with a variety of image elements, the live action footage and real photographs are assembled on screen as a thing in itself and as a thing in the composition (figure 17).

Figure 17: Hucklebarry Lain, Parallel (2010)

Composition

The compositional task for the author(s) of a visual music work involves making choices about the visual and music things themselves, as they are in each of their own domains, and also how the visual and music elements interplay and interconnect over time. An essential characteristic of a visual music piece is the unfolding of a temporal interaction between visuals and music. How this is structured over time is at the hands of the composer/ filmmaker / animator / visual music artist. Music composition strategies work well in a visual music work and many music composers who also create their own visuals have such a compositional aesthetic and can work with both visual and music material as being all the things available to them in their compositions. However, equally cinematic, fine art and animation techniques and computer animation techniques bring other approaches to composing with visual and music things. Having such knowledge, makes for a visual and its temporal evolution over time with music, that is like a moving art – fine art that has moved out of the constraints of the static frame and into the dynamic action of the temporal screen, cinema and animation uses its technology and knowledge of screen dynamics to explore the things of image themselves and to

compose with these basic building blocks of image, re contextualized when the music composition is as important as the images on screen. Video artists and particularly a new breed of video artist who use programmed and generated images and music or one or the other in their work bring another aesthetic dimension. The strategies of composition involve making decisions about the parametric control of the visual and music material. Setting up computer procedures and applying mathematical formulae to the creation of the building blocks of their material and to structuring the unfolding of the computer imagery over time. Regardless of the compositional aesthetic at work, the core common element of the compositional task in composing a visual music piece is structuring time and the core task is to make connections between visual and music material.

Visual music works are compositions. They may be presented in different formats, and with different processes, but essentially they are compositions, arising from deliberate aesthetic choices and intentions by the visual music artist/ collaboration. Music composition provides a model for structuring time and for composing with the abstract basic building blocks of music making. It has a long history of compositional techniques that can be used by the composer. Visual music works can re-work such music composition techniques into the visuals of a visual music composition, focusing on building compositions with the things themselves of image. Applying choice of colour, screen direction and dynamics, working with simple elements to complex image elements, rendering these elements over time, transforming and changing parts, re-mixing older parts in changed form, and basically crafting the simple things of image over time. Cinematic, fine art and computer art have all got a part to play in creating compositional approaches and strategies to a visual music work and in identifying the best principles for creating a visual music work. Visual music does not reference anything beyond itself. It exists as visuals and music in an inter-play of simple things, complex forms, from both visual and music material. The amount of contemporary practice in this area of visual and music interplay is fast beginning and, arguably, the start of the journey towards a shared language of composing with visual and music material. By suggesting that one such approach is to see and hear a visual music work as a composition of ‘the things themselves’, we give due recognition to the ‘things themselves’ that announce this exciting new visual art-form for and in the sonic arts.

Bibliography

Battey, B., 2009. Sinus Aestum. Video. Text and excerpt available at: http://www.mti.dmu.ac.uk/~bbattey/Gallery/sinus.html [Accessed 25 Aug, 2010]

Brougher, K., Mattis, O., Strick, J. And Wiseman, A., eds. 2005. Visual Music: Synaesthesia in Art and Music since 1900. Los Angeles: Museum of Contemporary Art; London: Thames & Hudson.

Cassidy, J., The Night After I Kicked It. Video. Excerpt available from: http://www.youtube.com/watch?v=4idsi29H8jY [Accessed 16 Aug 2010]

Center for Visual Music. Los Angeles. Website. Available from: http://www.centerforvisualmusic.org [Accessed 31st July, 2010]

Computer Music Journal, 2005. Winter 2005, Vol. 29, No. 4, Cambridge: MIT Press.

Counterlines, 2009. Video. Jaroslaw Kapuscinski, Javier Sanchez. USA: Intermedia Performance Lab. Available from: http://ipl.stanford.edu/Research.html [Accessed 20 Aug 2010]

Cuba, L., 1985. Film. Calculated Movements. Text available at: http://www.well.com/~cuba/CMovements.html [Accessed Aug 25 2010]

Detheux, J., 2010. Visual Abstract. Video. Excerpt available from: http://www.vudici.net/movies/Visual_Abstract/Visual_Abstract_final.html [Accessed 31 Aug 2010]

Eggeling, V., 1924. Film. Symphonie Diagnonale.

Elder, R.B., 2008. Harmony and dissent: film and avant-garde movements in the early twentieth century. Canada: Wilfrid Laurier University Press.

Fischinger, O., 1929-1934. Films. Studie Nr. 1 – Studie Nr. 13. Oskar Fischinger Filmography. Available online at http://www.oskarfischinger.org/OF_Filmo.htm [Accessed 21 Aug 2010]

Fischinger, O., 1947. My Statements are in My Work. In Art in Cinema catalog. San Francisco Museum of Art. Available from: http://www.oskarfischinger.org/MyStatements.htm [Accesed 20 Aug 2010]

Garro, D., 2009. Patah. Video. Text and excerpt available at: http://vimeo.com/14112798 [Accessed 20 Aug 2010]

Hébert, P., 2009. Triptyque. Film. Available at: http://pierrehebert.com/index.php/2009/09/02/138-triptyque [Accessed 25 Aug 2010].

Hébert, P., 2010. The idea of animation and instrumental expression. Available at: http://pierrehebert.com/index.php/2010/04/21/180-the-idea-of-animation-and-instrumental-expression [Accessed 25 Aug 2010] Translation from french, 2009.

Hyde, J., 2009. Seeing Sound Practice-led Visual Music Research Symposium. Bath Spa University, 19/20th Sept 2009. Available at: http://www.seeingsound.co.uk/ [Accessed 20 Aug 2010]

Hyde, J., MMus (Creative Sound and Media Technology). Available at: http://www.bathspampa.com/view-course.php?location=%2Fcourses%2Fcourse10&page=page2 [Accessed 05 August 2010]

IPL LAB, CCRMA. Research Available from: http://ipl.stanford.edu/Research.html [Accessed 19 Aug 2010]

Kapuscinski, J., 2001. Basic Theory of Intermedia Composing with Sounds and Images. Available from: http://www.jaroslawkapuscinski.com/writing.html [Accessed 06 Aug 2010]

Kapuscinski, J., Sanchez, J., 2009. Counterlines: Studies in Interfacing Graphic and Melodic Lines. In Proceedings of the International Conference on Computer Music, August 16-21, 2009, Montreal: ICMC 2009. Available from: http://www.jaroslawkapuscinski.com/pdf/studies-counterlines.pdf [Accessed 06 Aug 2010]

Kapuscinski, J., 2009. Juicy. Video. Excerpt available from: http://www.jaroslawkapuscinski.com/work-juicy.html [Accessed 06 Aug 2010]

Lain, H., Parallel. 2010. Video. Text and excerpt available at: http://huckleberrylain.net/index.php?/project/parallel/ [Accessed 16 Aug 2010]

LIA., 2008, Construction 76. Video. Excerpt available from: http://vimeo.com/1729281 [Accessed 16 Aug 2010]

Lund, C. and Lund, H., eds. 2008. Audio . Visual – on Visual Music and Related Media. Stuttgart: Arnoldsche.

Maxwell, S., 2006. All That Remains. Film. Available from: http://people.rit.edu/sampph/W_AllThatRemainsCompositions. DVD, 2007.