Tabula ex-cambio

Vincenzo LombardoCIRMA, Università di Torino

VR & MM ParkTorino, Italy

vincenzo@di.unito.it

Andrea ValleCIRMA, Università di Torino

Torino, Italyandrea.valle@unito.it

Fabrizio NunnariVR & MM Park

Torino, Italyfnunnari@vrmmp.it

ABSTRACTTabula-ex-cambio is an interactive installation that deliversvisual and audio content. It works as a sort of perpetual bil-liard game, powered by the data originating from the trendof a stock exchange index. The visual content is a real-timeanimated 3D computer graphics, that represents a billiardgame where each ball is associated with a stock in the index.The audio content is an electroacoustic music compositionfeaturing as many layers of sounds as stocks in the index.All the balls in the billiard are musical sources that playan iterated sequence of sounds while moving on the billiardtable; each sequence is altered in frequency by the trend ofthe related company index. The final delivered visual andaudio contents depend on the interaction with the user, whocan select a view/listening point on the billiard game. Thephysical installation consists of a deformed billiard structureconnected to a stele through cables; the stele is a vertical dis-play where the billiard game takes place. The visual displayrelies on the graphic engine Ogre, while the aural display isimplemented in SuperCollider. The installation was exposedin Shanghai, Beijing, Birmingham, and Terni (Italy).

Categories and Subject DescriptorsH.5 [Information Interfaces and Presentation]: Multi-media Information Systems; J.5 [Arts and Humanities]:Arts, fine and performing; I.3 [Computer Graphics]: Ap-plications

General TermsExperimentation, Human factors

Keywordsart installation, auralization, stock index presentation, real–time computer graphics

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.MM’09, October 19–24, 2009, Beijing, China.Copyright 2009 ACM 978-1-60558-608-3/09/10 ...$5.00.

1. INTRODUCTION

• tabula -ae f. [a board , plank; a draught-board; apainted panel; a painting; a votive tablet; a map; awriting tablet; a document]; in plur. [a record, register;a catalogue; an auction].

• ex–cambio: from changement, from exchange.

This paper is the description of the installation “Tab-ula ex-cambio”, in response to a call for artworks withinthe visual art project “Map Games: Dynamics of Change”.The project, started by the artist Varvara Shavrova and cu-rated by Varvara Shavrova, Feng Boyi, Monica Piccioni andRosario Scarpato, reflects about the rapid changes that af-fect the aspect and the dynamics of current China, and Bei-jing in particular. “Map games” refers to mental games thatcan track a personal perception of a changing town with re-spect to the three axes of history and memory, space andtime, language and communication.

The installation“Tabula ex-cambio”consists in a deformedbilliard table connected through cables to a stele that dis-plays a virtual billiard game driven by the trend of a stockexchange index (see Figure 1). The trend causes an accel-eration/deceleration of the balls with respect to an averageuniform speed; each ball is a music source too, that plays aniterated sequence of sounds. This rhythmic structure is pa-rameterized over the playing speed, that increases/decreaseswith the trend. The visitor can interact with the game byselecting the view/listening point that determines the vi-sual display as well as the render of the music composition.The selection occurs by placing a card, representing a ballof the billiard, on a sensible area (Figure 2). There are twoview/listening situations: a top view of the billiard game(no card on the sensible area), where ball motion and soundmixing are appreciated from a distance over the table (Fig-ure 3) and that represents the global activity of the stockexchange, according to the trend of the index; a run-aftercamera view, which focuses on one specific ball as selectedby the visitor (Figure 4), with a sound mixing that privilegesthe trailed sound source with respect to the other balls.

The installation aims at communicating the tension be-tween traditions, here symbolized by a wooden stele andthe billiards game, and modern lifestyle, here symbolized bythe electronic signal cables and the virtual billiards game(framed in the stele), powered by the stock market indextrend. The tension produces a number of dichotomies andconnections. The billiard game manifests a physical nature,represented by the real billiard table, while the abstract na-

1053

Figure 1: 3D render of the project for the installa-tion “Tabula ex-cambio”: overview.

Figure 2: Cards for selecting the view/listeningpoint. The cards contain an RFID sensor detectedby the antenna embedded in the sensible area, therectangular grey space in the picture.

ture of stock market index is revealed through the visual-ization/sonification on the stele. The billiard table is agedand subject to the competition strengths of the new soci-ety, that get at deforming its shape, thus altering the flatsurface. Such a deformation represents the marks that thenew lifestyle leaves in the social and architectural structure.The billiard table symbolizes, in its nobleness and popular-ity, such a structure, no more able to survive the anxietyof the new lifestyle. The traditional structure is threatenedby contemporary values, and it gets progressively deformedwhile trying to respond to new challenges. However, thenew society is the result of a historical process that startsfrom the traditions, as revealed by the cables that connectthe billiard table and the stele. Finally, the stele is a sur-face where the trend of economic power (the new god) iswritten down, but manifests itself again as a billiard game.So, the manifestation occurs in traditional forms, althoughmediated through a virtual world.

The currently represented stock exchange index consistsof 30 companies (and balls on the billiard table), with dataacquired through a simulation of a real index trend1. Theinstallation was exposed in April 2008 at the ContemporaryArt Museum of Shanghai for the art biennial “Animamix”,and from June 2008 to April 2009 at the OffiCina Exhibition“Map games”at the Today Art Museum in Beijing, Birming-ham Museum and Art Gallery (UK) and Terni InternationalCenter for Contemporary Art Ex-Opificio Siri (Italy). At the“Animamix” exhibition (Figure 5), the wooden stele was notcompleted yet, and the alteration of the billiard table was

1We had contacts with the official Italian market manage-ment body “Borsa italiana” for having the permission of op-erating with real data (the MibTel index actually consists of40 companies).

Figure 3: Billiard game top view (rotated of 90 de-grees with respect to the stele visualization).

Figure 4: Billiard game run-after view.

represented by a cover of scattered cloth and paper pieces.At the “Map Games” exhibition (Figure 6), the number ofcables is reduced and the deformation of the billiard table isrealized by a downhill table.

The rest of the paper illustrates the concept and designof the installation and then the software architecture forrealizing the visual and the aural components respectively.Now, we address some artworks that are related to “Tabulaex–cambio”.

2. RELATED WORKStock market is a typical example of a complex dynamic

system where the behavior of the whole system is radicallydifferent from the behavior of each component. Stock behav-ior generates large amounts of data varying at a fast rate.Visualization is fundamental not only in terms of represent-ing the overall market behavior over a given time-span, butalso as a way of exploring it in real-time. A large amount ofapplications have been developed to this aim, mainly basedon two–dimensional displays. As an example among themany possible, SmartMoney freely provides the well–knownapplication Map of the Market2, an on-line, interactive 2D

2http://www.smartmoney.com/map-of-the-market/

1054

Figure 5: The installation “Tabula ex–cambio” atthe “Animamix” exhibition in Shanghai.

Figure 6: The installation “Tabula ex–cambio” atthe “Map Games” exhibition in Birmingham.

map for visualizing market data, and more recently has de-veloped the Stock Market Radar application following anal-ogous criteria3.

Beyond the multimedia technical tools specifically devel-oped for helping traders, many artistic installations –inspiredby an info-aesthetic approach– have dealt with financial data,as the ever-changing flow of data not only challenges theartists to find apt solutions for managing sudden unforeseensituations, but also ensures a highly dynamic behavior tobe experienced by the users of the systems. We can no-tice that in these cases the displaying of information is justone aspect, or simply a point of departure for the aestheticstrategies. The immateriality of information is sometimescounterbalanced by physical setups. Andrea Nicolas Fischerhas realized Indizes4, a data sculpture in plywood visualiz-ing the stock market indices S&P 500, Dow Jones Industrial

3http://infosthetics.com/archives/2006/03/stock market radarsmartmoney data visualization.html4http://anfischer.com/indizes/

and NASDAQ in the year 2008 from January to November.While Indizes is evidently static, Plastic trade-off by GeraldNestler and Sylvie Eckerman is a light scuplture visualizingglobal financial markets where the real-time data of specificmarkets are translated into abstract light flows reflectingthe global financial system in a dynamic work of art5. Scal-ing up to architectural dimensions, The Source is an eightstorey high kinetic sculpture commissioned in 2004 to Grey-world6 as the new symbol for the London Stock Exchange.It is formed by a grid of cables arranged in a square: ninespheres are mounted on each cable and are free to move in-dependently up and down its length. In essence the spheresact like animated pixels displaying in real time the marketdata.

3D visualization often rely on virtual environments, likein Ecosystm by John Klima7, commissioned in 2000 by ZCMcompany: it uses financial data to drive a 3D environmen-tal simulation of global currency volatility fluctuations andleading global market indices, by branching trees and flock-ing birds. The viewers can interactively explore the envi-ronment with a joystick. Black Shoals Stock Market Plane-tarium8 is an animated night sky that is also a live repre-sentation of the world and the stock markets, with each starrepresenting a traded company. Fed by massive streams oflive financial information, the stars glimmer and pulse, im-mediately flickering brighter whenever their stock is tradedanywhere in the world. The work is intended to be expe-rienced via a distant observation, like a night sky. I DealSolution9 retrieves data from the New York Stock Exchangewebsite and uses the values of each stock to set the position,size, color, vibration and sound of a given object in the gen-erated 3D scene. While not interactive, it includes soundgeneration for data displaying.

The use of sound has enormously grown during the ’90s,leading to the establishment of sonification (that is, the pre-sentation of data using sound [9]) as an autonomous fieldof study. Many scholars have noticed that, while the de-scription of the relation among elements in a data structurebenefits from a visual displaying, the intrinsic temporal na-ture of audio makes it particularly apt to represent timeseries [4]. In particular, while a single time series is easilydisplayed visually (as noted by [14], “75% of all the graphicspublished [are] time-series”), the relation over several seriescan be more effectively grasped by auditory perception [11].Grasping benefits from the overall rhythmic organization ofthe resulting auditory streams [3]. Stock market data havebeen electively taken into account by studies in auditorydisplays (for an overview see [6], [10]), also through the de-velopment of dedicated softwares ([2], [10], [5]). All these ap-proaches share a common interests in real-time performance,as financial data flows continuously at a fast pace, and sonifi-cation (via real-time “auditory graphs”, [5]) can help tradersin promptly identifying critical situations. From an explic-itly artistic perspective, a real-time multimedia frameworkfor the displaying of stock-prices and other information in-cluding video projection and music has been proposed in2003 by UnitedVisualArtists for the World Tour of the trip-

5http://syl-eckermann.net/plastictradeoff/pto.html6http://www.greyworld.org/7http://www.cityarts.com/lmno/ecosystm.html8http://www.blackshoals.net/9http://www.idealsolution.eu/

1055

hop band Massive Attack10. The display of information onlarge screens was synchronized with live music (generatedby Massive Attack’s computers). In that case it was musicto control the flow of information.

3. CONCEPT AND DESIGN OF “TABULAEX–CAMBIO”

“Tabula ex-cambio” produces a visualization/sonificationof an index trend, with the billiard game metaphor thatprovides a visual/aural counterpart to the abstract numbersof the stock exchange index trend. The billiard metaphoroperates as follows.

The visual component is an interactive real–time CG ren-dering of a billiard game, based on a pool table without holesand powered by the index updates: each ball of the virtualbilliard is associated with a corporation stock of the index;balls move with constant speed; speed increases/decreasesin proportion with the trend of the corporation stock, so acorporation stock with a positive trend increases its speed,while a corporation stock with a negative trend decreases it.The visual component communicates the trend of the stockexchange, with a dynamics depending on the sign and thevalue of the considered index. The overall speed (of all theballs in the billiard) mirrors the general trend of the stockmarket, more dynamic with a bull market, more drowsy witha bear market.

The aural component is a music piece consisting of a num-ber of overlapping layers (strata, tracks) that are equal tothe number of balls in the billiard (or corporation stocks inthe index). Each ball of the billiard, also a sound source,produces one layer, while moving in the space of the virtualbilliard; layers overlap originating the piece; each layer isplayed with a variable rate that depends on the ball speed.So, the more effervescent is the stock market, the faster isthe playing speed of layers. The general trend of the stockmarket is sonified with the sounds of all the layers, fast rateswith a bull market, mixed rates with an average market,drowsy rates with a bear market.

The installation can be experienced through the individ-ual (audio or video) components or as a combined multi-media. Both the visual and aural components are interac-tive. Interactivity is determined by the positioning of theview/listening point (the two coincide), which is set by thevisitor of the installation. The default view/listening point isthe top view (Figure 3), with sound layers mixed in a stereopanorama. The visitor can set the view/listening point withrespect to a ball (or corporation stock) by selecting a place-holder, that represents a ball, and placing it on a sensiblearea (Figure 2). The application then focuses on the selectedball for both the visual and the aural renderings. This viewis called focus or run–after view. On the visual component,the effect is that a camera will shoot the selected ball by run-ning after it on its trajectory from a short distance (Figure4); on the aural component, the sound mixing emphasizesthe layer of the selected ball together with the layers of theballs encountered along the trajectory. The audio contentis actually determined by a distance function between theview/listening point and the sound sources: so, when theview/listening point is set to run after a certain ball, thelatter is the sound source that will be emphasized for dis-tance reasons, while other balls sounds will enter and leave

10http://www.uva.co.uk/archives/4

Figure 7: The information flow in the Tabula instal-lation.

the stereo panorama as they get closer and farther in thebilliard game, respectively.

The physical installation features a crippled pool table(pool table squashed at the front, kneeling down, dirty witha thick and oily grey cloth) and 60 meters of cables (con-nection from the stele base to the table through power andsignal cables). The distance between the two was designedto be about 15 meters. The stele features a 3–meter highwooden frame and a screen (the screen includes loudspeak-ers).

Now, we describe the software architecture and the tech-nical frameworks that account for the visual and the auralcomponents, respectively.

4. SOFTWARE ARCHITECTURE OF “TAB-ULA EX–CAMBIO”

The installation “Tabula ex–cambio” is based on a mod-ular architecture that consists of a controller, a visualizer,and two sound generators (see Figure 7).

The Controller takes as input the data from the stock ex-change index and the user’s focus (top view or run–aftercamera), updates the parameters of the physical dynamicsaccording to the recorded variations, and delivers the param-eters as well as the user’s focus to the Visualizer component.

The Visualizer is the core module for the simulation ofthe virtual billiard game together with its visualization. Thesimulation of the billiard game produces the directional speedsof all the balls in the billiard, and determines the collisionsthat occur at some point. The positions of the balls arecommunicated to the 3D graphic engine, together with thecamera motion (elaborated from the focus selection madeby the user). The data about the ball positions, speeds andcollisions are also returned back to the Controller, and thendispatched to the two sound generators. We can notice thatonly the Visualizer knows the balls positions and currentspeeds, that are detected by a snapshot after the data up-date of the virtual billiard game.

There are two modules for sound generation. The Ambi-ent sound generator accounts for ball collisions on the bil-liard table; the Sonifier produces the layered music given bythe overlapping sound sources associated with the billiardballs. The Sonifier maps a ball speed onto the frequency ofthe associated sound source and a ball position onto the spa-tial cues of the sound signal; the music execution delivered

1056

Figure 8: Software architecture of the Controllerand the Tarta4D environment. The latter imple-ments the Visualizer and the Ambient Sound Gen-erator.

also depends on the user’s selected focus (run–after camera),because only the balls/sources within a fixed audibility ra-dius that surrounds the run–after ball will be heard.

In the following, we describe the individual modules indetail.

4.1 ControllerThe Controller provides the mapping between the data

from the stock exchange index and the parameters of thephysical dynamics, and delivers such parameters to the Visu-alizer component. It also captures the user’s focus from thecard (actually a RFID sensor), to select the view/listeningpoint.

The Controller (Figure 8, left) maps the raise/fall value ofa corporation stock onto the speed of a ball. We mapped thestock range between -5% to +5% to a speed range between0.2 m/s to 1.7 m/s. At regular intervals of 0.5 seconds, theController reads the stocks and applies an impulse to thecorresponding ball in order to reach the desired speed. Theimpulse is applied along the speed vector in order to accel-erate or decelerate the ball without modifying its direction.This choice was made after the first tests, when we notedthat applying a random direction to the impulse created anunpleasant chaotic effect, thus missing the perception of thedynamics of the stock exchange activity. Only in the case aball is almost motionless (so the speed vector module is closeto 0 and the direction indefinite), the Controller computesa random direction on the horizontal plane.

The physical environment is setup with a number of spheres(one for each ball, 30 in the current implementation), onebox for the billiard plane, four boxes for the billiard tableborders and one infinite horizontal plane for the floor. Themass of each ball is set to a realistic value of 200 grams;the masses of the other objects are set to infinity (a specialvalue to state that objects are ideally fixed in space). Thisfeature, which helps in optimizing calculations, is exposedby the ODE engine. The gravity was set to the standardvalue of 9.81 m/s2.

The Controller was developed in Java.

4.2 VisualizationThe implementation of the Visualizer relies on the Tarta4D

Framework (Figure 8, right). The Tarta4D framework is de-veloped as part of the Enthusiasm project, an open–sourceplatform that supports the authoring of 3D real–time in-teractive virtual environments11. Tarta4D is a renderinglibrary, a 3D engine that offers high-level functionalities forbuilding applications based on real-time 3D technologies. Indetail Tarta4D offers the following functionalities:

• Graphics: Import of 3D objects authored with themost popular 3D authoring tools (such as AutodeskMaya, Autodesk 3DStudio, SoftImage XSI, and Blender),real–time 3D rendering (on top of DirectX or OpenGL),3D objects animation, automated animation blending;

• Audio: spatialized 3D audio, playback of pre-loadedsamples or in streaming mode;

• Physics: rigid bodies physical simulation, collision de-tection and report between physical objects;

• Miscellaneous: Multiplatform support (Windows, Linux,MacOS X), Simplified scene management (with In-put/Output support), Multi-thread support, C++ andJava APIs, Effortless integration in Java AWT/Swinginterfaces, GUI for scene authoring.

For its functionalities, Tarta4D relies on a number of open-source libraries: Ogre3D12 is the graphics library performingthe real-time 3D visual rendering; the Open Dynamics En-gine (ODE13) is used for the rigid-body physical simulation;OpenAL14 is used to render the real-time spatialized audiofor the case of sound design; inyXML15 is used to store andretrieve 3D scenes description in an XML format; SWIG16

is used to build the Java interface of Tarta4D.Many of the above mentioned Tarta4D functionalities are

provided by these underlying libraries. In the following wediscuss the functionalities added by the Tarta4D project.

Simplified scene management. In current high-level3D rendering engines (e.g., Ogre3D and OpenSceneGraph)the description of 3D scenes is based on the scene-graphparadigm OpenInventor [13]. The Tarta4D API (Applica-tion Programming Interface) exposes a simplified scene–treeapproach. A scene is defined as a tree of objects, camerasand lights. All the technical details about the reuse of re-sources (materials, geometries, skeletons, and the like) iscompletely automatized and hidden to the end programmer.A 3D Object in Tarta4D is a complex entity associated to avisual aspect, able to emit sounds, and responsive to gravityand collisions. A Scene is a container of objects character-ized by light condition (sun light and ambient light) and aRoot Object. Starting from the Root, the Objects are or-ganized as a hierarchical tree. Scenes can be saved to andloaded from files, in an XML format. In Tabula ex–cambiowe set up a scene with one object for the billiard, 30 objectsfor the balls, 2 lights (emitting a spot light from above thetable), and three cameras: one fixed camera shooting thetable from above, a second fixed camera shooting the tablefrom a side, and a third camera in motion that moves over

11http://enthusiasm.sourceforge.net12http://www.ogre3d.org13http://www.ode.org14http://connect.creativelabs.com/openal15http://www.grinninglizard.com/tinyxml16http://www.swig.org/

1057

the surface of the table running after the ball specified bythe Controller.

C++ Core and Java Binding. In the development ofreal-time interactive applications, where performance is of-ten a critical issue, C++ development is preferred to otherlanguages; on the contrary, most programmer–comfortablelanguages (e.g.: Java), featuring facilities such as automaticgarbage collection, sacrifice performances to favor of shorterdevelopment time. So, for the sake of performance, thecore of Tarta4D is implemented in C++. The Tarta4Dframework can be used in a C++ development environ-ment at its full potential. At the same time, all the pub-lic API is exposed in Java to favor fast development ofdomain specific applications. The aim is to keep all thetime-critical and CPU-consuming features inside the C++core, while reducing the development time of the generalstructure of new applications. As part of the Java binding,Tarta4D provides a graphical component (Tarta4DCanvas),showing the 3D rendering of Ogre3D, which seamlessly inte-grates into AWT/Swing based Java GUIs. The applicationTabula ex–cambio is completely developed in Java. TheTarta4D C++ core is loaded as one external DLL throughthe Tarta4DForJava wrapper. The Tarta4DCanvas, visu-alizing the Ogre3D rendering, is inserted into a borderlessSwing JFrame extended at full screen size.

Multi–threading support. Many of the libraries un-derlying Tarta4D are not thread-safe. For several techni-cal reasons, the calls to a library must be issued by thesame thread that initialized the library itself. This is prob-lematic when an application requires the integration of theTarta4D system with other multi-thread based systems, suchas many widget libraries used to build graphical interfaces.The Tarta4D API offers a thread–safe execution mode, es-pecially useful when Tarta4D goes through its Java binding.This feature allowed us to organize Tabula ex–cambio in aclean architecture with three different threads that handlethe user input, simulate the stock exchange, and update thevisual rendering, respectively.

Scene Authoring GUI. As part of the Enthusiasm plat-form, Tarta4D is provided with a GUI to compose scenesvisually. We used this interface to setup the scene of Tabulaex–cambio.

The Controller sees the Visualizer as a library throughthe Tarta4D Java wrapper. The 3D models of the billiardand the balls have been authored in Autodesk Maya andexported in the Ogre3D native mesh format. We editeda start scene referencing the billiard table and the balls.During the application startup the scene is loaded and theparameters of the physical simulation are setup.

The physical aspects of the billiard game simulation areaccounted for by the ODE engine. Collisions between ob-jects are treated with the ODE “Bounce Mode”, meaningthat colliding objects bounce off each other while losing apercentage of their speed. The speed reduction depends onthe type of the bouncing objects: ball vs. ball 0.9, ball vs.border 0.6, ball vs. plane 0.4 if the ball speed is >= 0.5 m/s(meters/second), 0.0 otherwise. In other words, in order tolimit the bouncing effect over the billiard plane, a ball will“stick” on the plane if falling from above at very low speed,which happens when a ball “re–enters” the game after hav-ing jumped out of the table because of its high speed. Thebouncing values have been set through observation trials,trying to obtain an interesting and amusing effect rather

than the perfect simulation of a real billiard. During thesimulation it might happen that a ball is thrown out of thebilliard, as result of hard collisions. When a ball exits thebilliard it will inevitably touch the (invisible) floor. In thiscase the ball speed is zeroed and its position is forced atabout 80 centimeters above the center of the billiard plane,forcing the ball to re-enter the game.

In closing this section, we address some issues about theusage of 3D real–time applications for artistic purposes. Ingeneral, the increasing diffusion of real-time 3D engines, es-pecially in the field of entertainment, made such technologyappealing for the creation of artistic installations. Real-time3D is even permeating the field of the Graphical User In-terfaces (see the Apple CoverFlow or the Microsoft Aerosystems), historically stuck to the metaphor of the two–dimensional desktop. However, the access to such technol-ogy is typically limited to the specific role of software de-velopers. Simplifying the use of 3D engines thus giving ac-cess to their functionalities to artists and non-programmersis still a challenge. The most diffused strategy to accom-plish this task is to provide the 3D engine with point-and-click authoring interfaces, where the task of coding the logicof an application is done through the visual programmingparadigm. This approach is extensively and proficientlyused in applications oriented to signal processing tasks [Eye-Web17, Max/MSP18]. But this approach is not suitable fordefining the behavior of interactive application, where com-plex data manipulation functions and logics are often re-quired. With the Visual Programming paradigm, definingcomplex functions through building blocks is a real pain, andoften requires the development of ad-hoc plug-ins (in C++).Processing19 follows a different approach: it aims at provid-ing the artist with a wrapper for an existing programinglanguage (Java), furnished with a light development envi-ronment and a set of every-day-use functions to manipulateimages, shapes and colors. Processing is very appreciatedby the artistic-technical community, but its support for 3Dtechnology is very primitive. It provides a basic mappingwith the OpenGL functions, forcing the developer to dealwith technical concepts, like the OpenGL state machine, andmath concepts like vectors, matrices and geometric transfor-mations.

4.3 AuralizationThe auralization process is performed by two separate

modules, the Ambient Sound Generator (ASG), for sounddesign, and the Sonifier, for the music execution process.

Ambient Sound Generator (ASG) The ASG, that isone of the components of the Tarta4D framework, relies onthe OpenAL library. It generates the sounds emitted in cor-respondence of collision events. Different sounds accountfor ball–to–ball, ball–to–border and ball–to–plane collisions(the latter type occurs when a ball re-enters the game). Thevolume of the sounds depends on the speed of the hittingballs. When a ball hits a border or a plane, we calculate themodule of its speed, multiplied by a constant (currently setto 2), and the result used as “gain” of the sound emission. InOpenAL, the gain is used as multiplier of the sample values.When two balls hit each other, we calculate the differenceof their speed vectors; so two balls hitting from opposite di-

17http://www.infomus.org/EywMain.html18http://www.cycling74.com/19http://www.processing.org/

1058

Sonifier Player

Sonifier Visualizer

state

velocity

focus

distanceArraypanArray

listener

"velocity"

"pan"

velocity

state "state"

oldListener "listener"

Sonifier ControllerSonifier

SchedulerOSC msg attributes notifications

Figure 9: The software architecture of the musicexecution component (the Sonifier).

rections produce a louder sound than two balls traveling onsimilar directions. Additionally, for a more pleasant effect,for the frequent case of ball–to–ball collisions, we introduceda variation in the auralization process by selecting randomlythe sound emitted out of four different sounds. Finally, inorder to reduce the overall noise created by such frequentevent, we had to filter out collisions between balls result-ing in a sound gain lower than 0.5. The two constants ofgain multiplication and gain threshold were selected via arefinement process after a number of tests.

Sonifier While the ASG is integrated in the Tarta4Dframework, the music execution component (the Sonifier)is a separate module. The Sonifier does not process directlythe stock exchange index trend, but receives the data fromthe Controller, together with information related to the vir-tual billiard model. In this sense, we can say that it sonifiesthe billiard dynamics: while adding a layer of mediationbetween source data and sound, the dependency of the au-ralizer on the visualizer tightens their relation.

The Sonifier features a modular architecture in order toensure the encapsulation of different functionalities: whileits main task is real-time audio generation and processing,it also handles network connection, manipulates the receiveddata in relation to mapping strategies, provides visual feed-back on its operations to the programmer. In order to bringtogether these issues, we turned to the SuperCollider20 (SC)application, as it features a high-level, object-oriented, in-teractive language together with a real-time, efficient audioserver. The SuperCollider language exhibits features thatare common to other general and audio-specific program-ming languages such as Smalltalk and Csound. They allowto generate programmatically complex GUIs and are struc-turally designed for networking ([15]). The whole systemheavily relies on the Observer design pattern ([8]) for eventhandling. The pattern allows loose coupling between –tospeak in SC terms– a “model” (the observed) and its “de-pendants” (the observers). The dependency mechanism isfundamental in allowing the maximum flexibility in the in-teraction with the system. The overall Sonifier architectureis represented in Figure 9 and features four components: aController, a Visualizer, a Player, and a Scheduler.

The whole sonification process can be described as follows.The general Controller regularly samples the state of the bil-

20http://supercollider.sourceforge.net/

8

13

193

(old 3)

Figure 10: Four balls on the billiard plane. Ball3 is the listener, its orientation is indicated by thedotted line. Distances and dislocations are indicatedby dash-dotted lines, arrows represent velocities.

liard. It sends via OSC the position and velocity of the balls,and also the focus to the Sonifier controller. The latter cal-culates pans and distances, i.e. information depending onthe focus and the balls. Then it notifies possible changes toits dependants. The Sonifier Visualizer displays (for testingpurposes) a 2D representation of the incoming data. TheSonifier Scheduler generates 30 layers of iterated sounds. It-eration rate is linked to a set of factors and the velocity ofthe ball related to each layer. The sounds are generated bythe Sonifier Player: it triggers a sample for each layer, andprocess it according to a cartoonified model of space, i.e. ituses distance and pan to simulate acoustic cues dependingon the relation between a listener and a sound source. Thespecial case of the absence of focus (top view) correspondsto the absence of a focused listener, and the lack of spa-tial data. The Sonifier Spatialiser of the Player is switchedoff: this creates a specifically different sound mood, strictlyanalogous to its visual correlates.

Sonifier Controller The Sonifier Controller componentis the interface to the overall Controller. It operates throughthree steps. The first step is the reception of the data fromthe general Controller through the network. The communi-cation is achieved via the OSC protocol [16]. The Controllersends three types of messages to the Sonifier:

• balls: an array with the positions of each ball over thebilliard plane expressed as a tuple

• focus: an identifier of the ball actually anchoring thepoint of view of the camera

• velocity: an array containing for each ball its linearspeed in a normalized range [0.0, 1.0]

Figure 10 shows a state of the billiard with 4 balls – 3, 8, 13,19 – for sake of simplicity, with 3 being the focus. For eachball, bold arrows represent velocity as a vector (velocity andorientation).

The balls array is [3 : [0.8774, 1.3245], 8 : [2.8321, 1.0378], 13 :[3.1751, 2.2221], 19 : [3.276, 1.0557]], the focus is 3, the ve-locity array is [3 : 0.5693, 8 : 0.9203, 13 : 0.4103, 19 : 0.8894](billiard table area is normalized to 5 × 3). The data arestored as attributes of the Sonifier Controller and representits internal state. The state includes other spatial-related

1059

samples

on HD

LowPass FilterReverb X

decay time ∝ cutoffFreq ∝ amp ∝ 1/

amplifier

Sample Player Spatialiser

rate ∝ 1/ Left

Right

distance

sample = balI ID

pan

Player

Panner

Figure 11: The player.

data that come from the physics simulation and are to beused in sound processing.

The second is the calculation of the internal attributesfrom the received data. The “balls” message updates two ar-rays: for each ball, the “distanceArray” contains its distancefrom the current focus (in the previous example [3 : 0, 8 :2.0231, 13 : 2.6202, 19 : 2.1163]); the “panArray” representsthe dislocation over the horizontal axis in the normalizedrange [−1, 1], where −1 indicates a source at the extremeleft and 1 a source at the extreme right (again, in the ex-ample above, [3 : 0.0, 8 : −0.9776, 13 : −0.3106, 19 : 0.1867],with balls 8 an 13 at the left of 3, and 19 at its right). The“focus” message updates the attribute “listener”, while theprevious value of “listener” is saved into the “oldListener” at-tribute. This information is necessary in order to calculatethe listener’s orientation. In Figure 10 the white circle rep-resents the oldListener value for ball 3. A constant attributeof the Sonifier Controller attribute is the audibility radius,that represents the listener’s perceptual threshold, i.e. thearea the determines which sound sources (balls) are part ofthe music composition. In the case of a “focus” view, soundsources enter and exit dynamically from the composition.

The third step is the notification to its dependants (Soni-fier Player, Scheduler, and Visualizer) that it has changedits internal attributes with the messages “balls”, “pan”, “lis-tener”, and “velocity”.

Sonifier Visualizer The Sonifier Visualizer provides vi-sual feedback to ease the designers’ and programmers’ work.Such GUI module receives a notification when the SonifierController updates its attributes. The GUI displays in a 2-D window the state of the balls and updates it each timea “balls” message arrives. The GUI also allows to Visualizethe audibility radius. As this information is related to thelistener, it is updated each time the “listener” changes. So,the programmer can better understand the relation betweenthe dynamics of the billiard and the resulting sound.

Sonifier Player The Sonifier Player is responsible forsound generation and real-time control. It defines a soundsynthesis device, with two modules (Figure 11).

The first module is a variable-rate Sample Player. In fact,the first aspect of the mapping strategy is the associationof a sample to each ball, and consequently to each corpora-tion stock. Samples are selected from instrumental soundsto have a clear event-like dynamic profile and to be as easilyas possible discriminable one form each other. The use ofevent-like, instrumental samples is aimed at providing per-ceptual and cultural cues helping the user to discriminateamong samples. Discrimination in parallel streams is in-deed one of the main issues in sonification as a consequence

of non–orthogonality of auditory dimensions, i.e. the factthat auditory dimensions are not mutually independent [1].

The second module is a processing unit Spatializer, in cas-cade with the Sample Player. While the Sample Player isresponsible for the generation and processing of each (basic)sound event, the Spatializer (and also the Scheduler below)operates a variation on sample playing through the mappingof spatial (and rhythmic, in case of the Scheduler) cues. TheSpatializer unit is the main responsible for the data sonifi-cation, taking into account its semantics. We adopted anunmarked position with respect to the spatialization of themusic execution in the top view case (all distance and panarray values are set to 0). On the contrary, in the focusview, the relation between the “listener” and the space (i.e.,the billiard plane where the balls are in motion) makes useof the acoustic spatial cues, a pretty natural solution formapping the billiard game onto the music execution. TheSpatializer (Figure 11) simulates, in a ”cartoonified” way,the aural perspective of the selected ball (that is, the lis-tener). Rocchesso et al. [12] have proposed ”cartoonifica-tion” techniques for sound design, i.e. simplified models forthe creation of sounds related to physical processes and spa-tial cues. The cartoonification process starts from an analy-sis of the physical situation and simplifies it, retaining onlythe perceptually and culturally relevant features. The Spa-tializer aims at cartoonifying monaural distance cues anddislocation cues [7]. Distance cues result in changes of re-verb, low–pass filtering, amplification, and playing samplerate. The decay time of the reverb and the cutoff frequencyof the lowpass filter are proportional to the distance, whilethe amplifier coefficient and the playing rate are inverselyproportional to distance. So, on the one hand, a sample isheard –respectively– more reverberant, duller, and quieter,when the source moves away from the listener, and the de-crease of playing rate implements a crude but clear Dopplereffect, thus contributing to shift the spectral energy to lowerregions. On the other hand, a sample is heard –respectively–drier, brighter, and louder, when the source moves towardsthe listener, with the increased playing rate realizes an op-posite Doppler effect, thus shifting the spectral energy tohigher regions. Concerning dislocation, the pan is used tofeed a two channel equal power pan, where −1 places thesound source at the extreme left and +1 at the extremeright, with 0 indicating a perfectly frontal source.

Scheduler While the Spatializer modifies the sound prop-erties according to spatial cues, the Scheduler organizes themusic events in time, in particular controlling the density ofmusic events.

The scheduling model is based on a minimalistic, canon-like technique. There are as many layers as balls in thebilliard, and each layer simply consists in the repetition ofthe sample associated to that ball (see Figure 12).

Each layer is characterized by a fixed period factor, whichis a multiple of a time unit (empirically fixed to 0.1 seconds).The period states the temporal interval between two subse-quent onsets of the sample playing. So, the sound source atlayer 1 plays every 0.1 seconds, the layer 2 plays every 0.2seconds, the layer 20 plays every 2 seconds; low–numberedlayers feature more frequent events than higher–numberedlayers. Consider that the geometric organization of the gridresults from the number of layers and the inner periodicity ofthe layers; however, notice that the same organization maynot be (and actually is not, in our case) perceived acous-

1060

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

Figure 12: Temporal proportions among layers.

tically, since we have no constraints on the sample length,and there is much and irregular overlapping of sound emis-sions during the execution. In general, since periods shorterthan sample lengths lead to overlap the same sample severaltimes, the artist preferred to use longer samples in longerperiods.

Moreover, during the execution, the basic organization ofevent succession in each layer, illustrated in Figure 12, isscaled for the velocity currently associated with the corre-sponding ball. This affects the event density. The durationof the period is scaled by a velocity factor that is inverselyproportional to the velocity associated to the ball of thelayer, so periods are shorter for fast balls (density increases)and are longer for slow balls (density decreases); the fastera ball rolls, the faster the associated sample is scheduledfor playing. In order to avoid a total asynchronism amonglayers, the velocity factor is quantized: in this way, the lay-ers are anchored to a temporal grid, even if very fine (gridstep empirically determined to 0.125 seconds, so the mini-mum period can be shortened to one eighth of its length).The iterative yet ever-changing soundscape is intended asan aesthetic representation of the hectic/flabby activity ofthe stock market.

In Figure 13 there is a plotting of the audio events asresulting from one run of the installation. The arithmeticprogression of the time grid is constantly deformed by thevelocity associated to each layer and by the filtering operatedby the listener. As a result, proportions only define averagedensity for each layer. Even if not apparent from the plot, allthe events are still scheduled on the basis of the common gridstep. The complex, noisy and fragmented result depends onthe large number of layers.

5. CONCLUSIONSThis paper has presented the installation Tabula-ex-cambio,

that features a multimedia component that represents a vir-tual billiard game powered by the data originating from thetrend of a stock exchange market index. The installationwas exposed at the Contemporary Art Museum of Shanghai

for the art biennial “Animamix” (April 2008), and at the Of-fiCina Exhibition“Map games” in Beijing, Birmingham(UK)and Terni (Italy) (June 2008 to (April 2009).

Tabula Ex-cambio aims at integrating a 3D audiovisualrepresentation of stock market data with a physical object,acting as a symbol of tangible effects of information, andto complement visual information with data–driven music.More, the resulting installation allows for an interactive ex-ploration of data from the view/listening point. As for thevisual metaphor, we chose to represent stock trade througha billiard. A common subject in video games, billiard isstrongly recognizable, both from a concrete point of view(the billiard is a well–known cultural item) and from an ab-stract one (a billiard is in itself the physical representationof an idealized geometric world made of spheres on a plane).The paper has illustrated the concept, the design and theimplementation of the installation. We have also describedthe computational frameworks the installation relies upon,namely the graphic engine Ogre, the module Tarta4D, andthe Supercollider environment.

The paper has also illustrated a number of works in theartistic and technical realms that are related to our instal-lation. Stock exchange index (that collects a selected set ofsignificant corporation stocks) is one of the major topics ofnews reports and a topic for information visualization (seeSection 2). Many contemporary installations dedicated tostock exchanges are concerned with physical setups. Withrespect to this, the 3D virtual billiard acts as a counter-part of the physical billiard: it can be said that it medi-ates between the physicality of the real billiard table, andthe pure immateriality of the numbers. This immateriality,while solely consisting of flows of information, still is capableof generating tangible, physical effects on real life.

6. ACKNOWLEDGEMENTSWe thank the artists Paolo Armao, Pino Cappellano and

Andrea Gotti, who are part of the team of the installationTabula–ex–cambio. In particular, the credits for the instal-lation are: Andrea Gotti (billiard/stele concept and physical

1061

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

Figure 13: A plotting of the audio events as resulting from one run of the installation.

realization), Vincenzo Lombardo (concept and direction forthe stele virtual contents), Andrea Valle (sonification), Fab-rizio Nunnari (software development), Pino Cappellano (vi-sual artist and CG animator), Paolo Armao (foley artist).Tabula–ex–cambio is supported by CIRMA, Universita diTorino, and ASA–Lab, Virtual Reality & Multi Media Park.The web site of the installation is http://www.cirma.unito.it/Tabula–ex–cambio.

7. REFERENCES[1] S. Barrass. Auditory Information Design. PhD thesis,

The Australian National University, Acton, ACT,1997.

[2] O. Ben-Tal, J. Berger, B. Cook, M. Daniels, andG. Scavone. SonART: The sonification applicationresearch toolbox. In R. Nakatsu and H. Kawahara,editors, Proceedings of the 8th InternationalConference on Auditory Display (ICAD2002), Kyoto,Japan, 2002. Advanced Telecommunications ResearchInstitute (ATR), Kyoto, Japan.

[3] A. Bregman. Auditory Scene Analysis. The MITPress, 1990.

[4] E. Childs. Auditory graphs of real-time data. InE. Brazil, editor, Proceedings of the 11th InternationalConference on Auditory Display (ICAD2005), pages402–405, Limerick, Ireland, 2005. Department ofComputer Science and Information Systems,University of Limerick.

[5] E. Childs. Auditory graphs of real-time data. InProceedings of the 11th International Conference onAuditory Display (ICAD2005), pages 402–405,Limerick, Ireland, 2005.

[6] F. Cifariello Ciardi. sMax: A multimodal toolkit forstock market data sonification. In S. Barrass andP. Vickers, editors, Proceedings of the 10thInternational Conference on Auditory Display(ICAD2004), Sydney, Australia, 2004. InternationalCommunity for Auditory Display (ICAD).

[7] F. Fontana and D. Rocchesso. The Sounding Object,chapter Synthesis of distance cues: modeling andavalidation, pages 205–220. Edizioni di MondoEstremo, Firenze, 2003.

[8] E. Gamma, R. Helm, R. Johnson, and J. Vlissides.Design Patterns: Elements of Reusable ObjectOriented Software. Addison-Wesley, 1995.

[9] T. Hermann. Sonification for Exploratory DataAnalysis. PhD thesis, Technischen Fakultat derUniversitat Bielefeld, Bielefeld, 2002.

[10] P. Janata and E. Childs. Marketbuzz: Sonification ofreal-time financial dataa. In S. Barrass and P. Vickers,editors, Proceedings of the 10th InternationalConference on Auditory Display (ICAD2004), Sydney,Australia, 2004. International Community forAuditory Display (ICAD).

[11] K. V. Nesbitt. Comparing and reusing visualisationand sonification designs using the ms-taxonomy. InS. Barrass and P. Vickers, editors, Proceedings of the10th International Conference on Auditory Display(ICAD2004), Sydney, Australia, 2004. InternationalCommunity for Auditory Display (ICAD).

[12] D. Rocchesso and F. Fontana, editors. The SoundingObject. Edizioni di Mondo Estremo, Firenze, 2003.

[13] P. S. Strauss and R. Carey. An object-oriented 3dgraphics toolkit. SIGGRAPH Comput. Graph.,26(2):341–349, 1992.

[14] E. R. Tufte. The Visual Display of QuantitativeInformation. Graphics Press, Cheshire, Conn., 2nd ed.edition, 2001.

[15] S. Wilson, D. Cottle, and N. Collins, editors. TheSuperCollider Book. The MIT Press, Cambridge,Mass., 2008.

[16] M. Wright, A. Freed, and A. Momeni. Opensoundcontrol: State of the art 2003. In F. Thibault, editor,Proceedings of the 2003 Conference on New Interfacesfor Musical Expression (NIME-03), 2003.

1062