From Technologies of Representation to Technologies of Performance

Sha Xin WeiTopological Media Lab, Concordia University, Canadasha@encs.concordia.ca

1. The problem of representation, and the performative turn.

Typically in our practices as designers, architects, scientists, or artists, we build and respect walls between designer and maker, maker and user, analyst and analysand, and spectator and actor. Guy Debord argued in the Society of Spectacle that it was the separation of life from its products which led to much of the banaliza-tion and anestheticization of contemporary social experience. This criticism echoed many critiques of life in industrial, and now technologically mediated society. A more contemporary, post-marxian critic, for exam-ple, could argue the same for the separation of life from its affects.

In this very large domain, one of the most fundamental elements of this critique was the problematization of systems of representation. To pick just one prominent example, Ludwig Wittgenstein famously demolished the presumed universally stable, intrinsically and abstractly analyzable meaning of words and sentences. In the Philosophical Investigations, capping an extended series of paragraphs in which Wittgenstein demon-strates the implausibility of the thesis that the meaning of language can be captured in a system of rules or can be determined by a rule-based procedure, he writes: “To understand a sentence means to understand a language. To understand a language means to be master of a technique.”1 The first statement says that meaning is not entirely localizable. And the second, even more significantly, says that meaning comes from use and practice.

Despite the ubiquity of the paradigm of the Graphical User Interface, going graphical does not yield intrin-sic, context-free meaning. Wittgenstein extends his analysis to the interpretation of graphic pictures as well. In §454, he writes:

How does it come about this arrow >>>-----> points? Doesn’t it seem to carry in it something besides itself?—“No, not the dead line on paper; only the psychical thing, the meaning, can do that.” -- That is both true and false. The arrow points only in the application that a living being makes of it.2

Just as in the case of verbal speech, the meaning of a graphic sign lies in its practiced, and collective use.

Of course, this is only one, but sharp, example of the modern problematization of representations in media ranging from writing to photographs, film, and digital video. The larger problem naturally leads us to ques-tion whether any technology for the creation and diffusion of representations could possibly be adequate to our social or aesthetic practices.

Many of our computational tools for design are used explicitly as technologies of representation. Can com-putational instruments do something other than separate designers from the products and the experiences of their designs? As the architectural experimental group Adaptive Actions put it in their documentation:

“Architects often prefer photographing/showing buildings at the height of their glory: when the presence of time is imperceptible and user-trace absent. Some architectural agencies even con-trol representation, allowing circulation and posting of approved images only. 'Now' is the mo-dus operandi – priority goes to the image of the building in the present and very little concern to its progression, to the future. Much emphasis is given to what must be photographed, hon-

oured, recorded and published in magazines rather than to users' adaptation of space and ap-propriation in various forms.” 3

Of course this is not an isolate or new sentiment in the critical history of the relationship between architec-ture and technology. But what about the even more fundamental gap between our bodies from our cogni-tive models?4 The gap I refer to is not so much the old body-mind dualism, but the inadequacy of any theo-retical schema to pragma, to which Wittgenstein pointed. The turn to the body implies a deeper turn to ac-tion, and to performance, those complexes of related, irrepeatable and intentional actions that create an event. Less anthropocentrically, we can regard performance as the primordial flow of matter holding in sus-pension the problematic distinction between living and the inert. This materialist ontology has a long history that ranges from Heraclitus to Maturana and Varela, and Karen Barad. Barad describes succinctly the mod-ern critique of representationalism and the move toward performance in a chapter on material-discursive practices from her book Meeting the Universe Halfway. Barad writes:

“Is it not, after all, the commonsense view of representationalism -- the belief that representa-tions serve a mediating function between knower and known -- that displaces a deep mistrust of matter, holding it off at a distance, figuring it as passive, immutable, and mute, in need of the mark of an external force like culture or history to complete it?” 5

Barad continues: “A performative understanding of discursive practices challenges the representationalist belief in the power of words to represent preexisting things.” 6 Barad refers to words, but my point, along with Wittgenstein and Barad herself as well, pertains to all representations. “The move toward performative alternatives to representationalism shifts the focus from questions of correspondence between descriptions and reality. (e.g. do they mirror nature or culture?) to matters of practices, doings, and actions.”7

2. The performative turn and technologies of performance

Consider a software application like Photoshop. Here the creator makes an image that another person sees as a finished work whose form and content cannot be altered by that spectator’s activity. The same with Adobe After Effects, Auto-CAD, and Rhino. With Adobe After Effects, the creator makes transformations on the video material, but each time the final video is viewed, the same transformations occur at the same place in the video. Of course there is the pedantic case of the creator as spectator. But my point concerns whether the intended audience regards the created entity as it is shaped by the creator, or whether the audi-ence can re-shape the entity by interacting with that entity. Whether the entity is itself made of sound or video or plastic, the result is still a representation of some other thing.

However, there is a whole other category of technologies oriented to live performance and realtime interac-tion between the media entity and the spectator. With realtime video processing software like NATO, VVV, PD, or more professionally, Max/MSP/Jitter, a composer creates not a linear sequence of images and sounds, nor even a set of discrete pieces of media that can be permuted and selected according to some “user input” but a set of conditions for the spatially and temporally continuous modulation of streams of video and sound simultaneous with the gesture of the person(s) engaged with the responsive media.8 The analogy is a musi-cal instrument like a violin, or the bodily apparatus of a singer.

A technology of performance does not have to be immaterial or only concerned with the synthesis and modulation of time-based media. In fact, to identify the technology of the ephemeral performance as ephemeral media processes would be to commit a grammatical error of a Wittgensteinian sort. Wittgenstein famously said: “Don’t regard a hesitant assertion as an assertion of hesitancy.” Similarly we should not con-fuse a technology that represents performances with a technology that mediates performance.

Now, having sketched the distinction between computational technologies not as technologies of representa-tion but as technologies of performance, let us pause and ask why should we make this move? I suggest that

Sha, Technologies of Performance Harvard Critical Digital 2009

2 / 8

this is one way to claim, or reclaim, something of an ethico-aesthetic relation to our work. Do designers inhabit their own products? Should they? The economics of design process as it stands, based on a se-quence of transformations of representations powered in each stage by commitments of ever larger amounts of capital, makes it difficult to make a living sketch of an environment that could be inhabited by its putative inhabitant-creators. Nonetheless we can ask: If we had to inhabit the environments that we design, would we design them quite differently?

Returning to questions of technique, what would it mean to make a sketch? Perhaps it would help to make explicit another inspiration for the sort of strategy that I am suggesting with responsive environments, regard-ing theater as a mode of experiential research. In his landmark book, Towards a Poor Theatre, Jerzy Gro-towski writes:

“The Rich Theatre depends on artistic kleptomania, drawing from other disciplines, constructing hybrid- spectacles, conglomerates without backbone or integrity, yet presented as an organic artwork. By multiplying assimilated elements, the Rich Theatre tries to escape the impasse pre-sented by movies and television. Since film and TV excel in the area of mechanical functions (montage, instantaneous change of place, etc.), the Rich Theatre countered with a blatantly compensatory call for “total theatre." The integration of borrowed mechanisms (movie screens onstage, for example) means a sophisticated technical plant, permitting great mobility and dy-namism. And if the stage and/or auditorium were mobile, constantly changing perspective would be possible. This all nonsense. No matter how much theatre expands and exploits its mechanical resources, it will remain technologically inferior to film and television. Conse-quently, I propose poverty in theatre. We have resigned from the stage and auditorium plant: for each production, a new space is designed for the actors and spectators.

Thus, infinite variation of performer-audience relationships is possible. The actors can play among the spectators, directly contacting the audience and giving it a passive role in the drama .... Or the actors may build structures among the spectators and thus include them in the archi-tecture of action, subjecting them to a sense of the pressure and congestion and limitation of space....”9

Our situation is not Grotowski’s, but there are lessons to be drawn. Grotowski’s actors economically achieve their effects with greater symbolic and physical intensity than what can be achieved by representational me-dia with a hundred times the technical investment, by training. However, pushing the argument of the pre-vious paragraph to its limit, the designers of an experimental architectural event may be both actors and spectators in that event, so some actors in the event will be non-expert, or non-rehearsed. Therefore we do not have his pure condition. I suggest that we take Grotowski’s approach more symmetrically (in Barad’s sense) between the set’s materials and the human inhabitants of an event.

One of the Poor Theater’s tactics is for the actors to build a set’s structures in the course of an event, rather than introduce elaborate, pre-constructed sets. And to leverage and engage the imagination of the inhabi-tant, every prop must be as flexibly re-signified as possible. For example, a broomstick can become horse, a dance partner, a crucifix. This argues for media that can be fluidly re-shaped by the inhabitant rather than prepared with an elaborate, pre-composed syntax and semantics. With responsive media techniques, pro-jected image becomes chiaroscuro illumination; a table becomes a drum. (See Figure 2.)

What of the actors? In an environment built for “everyday life,” we may not always have actors expertly re-hearsed in “performing” an event. But the expertise that inhabitants can draw on is the deep and non-articulated sediment of all the corporeal intuitions built over a lifetime from birth, all the intuitions that every body brings into an event. And just as a physical set could be re-configured in the course of an event, so could the performative structure, the roles and relationships between the inhabitants be re-configured as well. Some inhabitants may be rehearsed and others not. Moreover, what we construe as event in every-day situations may not be restricted to one marked period of time, and certainly may not assume pre-constructed conversational templates. Instead, one could study how people co-habit an event, how we en-

Sha, Technologies of Performance Harvard Critical Digital 2009

3 / 8

train or engage one another in common fields of matter and media. This provides a profound motivation for working with non-figurative, responsive media that bear some of the manipulable, palpable qualities of physical matter. To support that, we would need a palette of computational media that permits manipula-tion as freely as ink or sand, yet affords practice with more refined effect.10 We will return to this in the next section.

3. Responsive environments as a technology of performance

Of course, the temporal and energetic scales of media technologies typically do not reach the much greater scales implied in architecture. Hence the interest in so-called virtual reality visualization systems in some technology-driven approaches to architectural research. But as we have seen, there is a technological hu-bris in attempting to entirely replacing the perceptual field with a synthetic one.

I propose that designers of built space using responsive media (rather than “interactive” media) could take advantage of certain powerful affordances of emerging technologies of performance for the creation of ex-perimental events in real-time, responsive environments.

Many different environments have been proposed and built over the past 50 years. For example, Gordon Pask and Robin McKinnon-Wood’s MusiColour system (1953) created light-fields and sound-fields in a physical space that reacted to the activity of the inhabitants’ physical activity according to some response logics.11 The key distinction here is not so much the sorts of technologies employed but the intent with which they are combined and used, and by whom. Against the homogeneously electronic cybernetic sys-tem, I ask whether we can institute a practice of experimental sketching of first-person experience in built environments that capitalizes on responsive media techniques as well as all the technologies of perform-ance, whether computational or not. It seems that, given the conditioning effects of any technology, a de-signer would do well to use poetic economy and power with computational means as well.

The key differences between such environments and a virtual reality system are that there is not necessarily any attempt to represent some structure that is “elsewhere”; nor is there the intention to wholly replace the “natural” sensory experience by the synthetic. Instead, we start with the full, thick, embodied experience of everyday materials and props, and augment it with the judicious insertion of responsive media that would sustain the experiential conditions required for an embodied engagement with an experimental configura-tion of architectural elements.12

I close with some more contemporary examples of responsive environments ranging from the TGarden play space, to the Ouija movement experiments on intentional and collective gesture, and on-going research with the Ozone media choreography system.13

In 1997, I proposed a responsive environment called TGarden (Figure 1) with the following characteristics:• Inhabitants’ engage each other not via explicit verbal communication, but via material fields (air, fabric,

etc.) and temporal textures of structured light and sound;• The media fields are dense enough to put in play where a body ends and the world begins;• The states of the event evolve and superpose continuously according to quasi-physical dynamics, a chal-

lenging alternative to the discrete finite state automata model of digital computation.Together with the art groups Sponge and FoAM, prototypes (TG2001) and related environments (txOom) were exhibited in 2001-2002 in Linz, Rotterdam, Athens, Torino.

Sha, Technologies of Performance Harvard Critical Digital 2009

4 / 8

Figure 1. TGarden, Ars Electronica and Dutch Electronic Art Festival 2001.

In 2001, I established the Topological Media Lab to develop certain strands of this work much more system-atically. Concisely put, the TML is an atelier-lab experimentally studying distributed modes of gesture, agency, and materiality from phenomenological as well as material performative perspectives.

The TML’s methods include: wireless sensor networks to incorporate any sort of physical signal using pattern tracking and sensor feature extraction techniques; calligraphic video -- computationally synthesized video as structured light, temporal textures via quasi-physical simulations; gestural sound -- realtime sound re-synthesis; wearables and active textiles, together with the traditional crafts of theater: lighting, scenography, movement, costume, elements of musical or sonic art. All the media used or created in the atelier is de-signed to respond in realtime to gesture and activity. And the approach strategically sidesteps cognitivist approaches to “user interaction” or “user experience.”

Over the last five years, the TML has honed these experimental techniques by working with choreographers, realtime video performers, sound artists and musicians. Rather than just building stand-alone artworks, however, the TML has built experimental apparatuses for studying gesture, agency, and materiality.

One of the most sustained of these experiments was the two month-long Ouija residency (Figure 2) in the Hexagram-Concordia research blackbox -- a theatrically equipped working space of 15m x 15m x 7m, de-signed to explore the following questions refined out of our experiences building responsive environments:• When is a gesture intentional, and when is it accidental? How would a human or machine system distin-guish between such grades of intentionality?• When does a set of actions constitute a collective gesture, rather a set of autonomous gestures? How

would a human or machine system distinguish collective vs solo gesture?The ancillary question alongside these principal phenomenological questions was: What difference does responsive media make in these situations?

Sha, Technologies of Performance Harvard Critical Digital 2009

5 / 8

Figure 2. Ouija experiment: shaping live-processed video as structured light.

An experienced choreographer and researcher devised a series of movement experiments for two to six dancers, plus zero to six non-prepared participants. These experiments were described in terms familiar to the dancers as structured improvisation. For the experimental apparatus, the TML created responsive fields with projected video, sound, and lighting, plus a few sensor-augmented furniture and props. We are ana-lyzing the results of the four weeks of recorded activity from different perspectives, and will report on that in other venues.

Based on the 8 years of experience with technologies of performance, over the past two years the TML has begun to host architectural research. This research ranges from a year-long studio (with an architectural theorist/practitioner and 3 graduate architecture students) focussing on a contested part of Montréal called Griffontown, through a several waves of immigration and industrialization over three epochs of energy economies since the early 1830’s; to temporary actions and urban installation-interventions by affiliates of the atelier.

Over the coming years, the atelier-lab is oriented to designing work in the built environment both as sited experiments in gesture and temporal texture, but also as more durable conditioning of events in public space. We expect that in order to pursue such work with any degree of phenomenological rigor and poetry, we will need to reconstruct technologies around human, infrastructural and environmental activities that performatively create events in the built environment. Taking this thesis seriously suggests that we will need to construe technologies of performance more ambitiously than has been the case in media, and in design, especially if we pursue the implications of blending designers’ and inhabitants’ agencies in a responsive built environment. Not least is the fact that every opening created by interactive, and now, responsive tech-nologies of performance also presents another opportunity for legal, socio-economic discipline expressed as policies implemented in the material as well as computational infrastructure of our built environment.

4. Acknowledgements

I thank the 60 artists, scholars, researchers and students who have been affiliated with the Topological Media Lab’s experiments, particularly the present affiliates of the atelier-laboratory: Michael Montanaro; the Ozone

Sha, Technologies of Performance Harvard Critical Digital 2009

6 / 8

research group: Harry Smoak, Tim Sutton, Morgan Sutherland, Michael Fortin, Jean-Sebastien Rousseau; plus the Dedale architecture studio: Patrick Harrop, with Gregory Rubin, Candace Fempel, Evan Marnoch.

References Barad, Karen Michelle. Meeting the Universe Halfway : Quantum Physics and the Entanglement of Matter

and Meaning. Durham: Duke University Press, 2007.

Debord, Guy. The Society of the Spectacle. New York: Zone Books, 1994.

Grotowski, Jerzy, and Eugenio Barba. Towards a Poor Theatre. 1st Routledge ed. New York: Routledge, 2002.

Haque, Usman. "The Architectural Relevance of Gordon Pask." In 4dsocial: Interactive Design Environments, Lucy Bullivant (ed.), 54-61. London: John Wiley and Sons Ltd., 2007.

Sha, Xin Wei. "The TGarden Performance Research Project." Modern Drama: world drama from 1850 to the present 48.3, Fall, Special Issue: Technology (2005): 585-608.

Sha, Xin Wei. "Poetics of Performative Space." AI & Society 21, Birthday Issue: From Judgement to Calcula-tion. 4 (2007): 607-24.

Simondon, Gilbert. Du Mode D'existence Des Objets Techniques. Paris: Aubier, 2001.

Sponge, TGarden: http://topologicalmedialab.net/xinwei/sponge.org/projects/m3_tg_intro.html

Topological Media Lab, Ouija: åhttp://www.topologicalmedialab.net/joomla/main/content/view/155/11/lang,en/ .

-----, Media Choreography: http://www.topologicalmedialab.net/joomla/main/content/blogcategory/5/21/lang,en/ .

Wittgenstein, Ludwig, and G. E. M. Anscombe. Philosophical Investigations: The German Text, with a Re-vised English Translation. 3rd ed. Malden, MA,: Blackwell Pub., 2003.

Sha, Technologies of Performance Harvard Critical Digital 2009

7 / 8

1 See Wittgenstein, L, and G. E. M. Anscombe. Philosophical Investigations, 3rd ed. Malden, MA.: Blackwell Pub., 2003, § 199.

2 Ibid. § 454.

3 See Prost, J-F, “Adaptive Actions,” forthcoming, AI & Society special issue on Soft Architecture, Sha, X.W. (ed.) Berlin: Springer-Verlag.

4 It is telling that in a recent talk about “embodied cognition,” and the activity of throwing a ball, the speaker showed FMRI images of the brain’s neural activity while the subject threw a ball, but no images of the whole body playing in its environment.

5 See Barad, K., Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham: Duke University Press, 2007, p. 133.

6 Ibid.

7 Ibid., p. 135.

8 I introduce the term, responsive media, to distinguish this sort computational media from so-called “inter-active” software, which is built around the Shannon-theoretic model of communication in which a sender encodes a message into a packet which is transmitted over a channel to a receiver who in turn decodes it. Interactive software implicitly assumes a turn-taking model of conversation in which only one speaks at a time, when in fact, most experienced events are densely full of concurrent processes.

9 See Grotowski, J, and E. Barba, Towards a Poor Theatre, 1st Routledge ed. New York: Routledge, 2002, p. 20.

10 In a word, virtuosity.

11 See Haque, U., "The Architectural Relevance of Gordon Pask," in 4dsocial: Interactive Design Environ-ments, Lucy Bullivant(ed.), 54-61, London: John Wiley and Sons Ltd., 2007, p. 56.

12 By element, I am deliberately drawing on Gilbert Simondon’s account of the evolution of technical ob-jects. See Part 1 of Simondon, G., Du Mode D'existence Des Objets Techniques, Paris: Aubier, 2001.

13 See the Showcase and Research links from http://topologicalmedialab.net, and search for these projects by name: TGarden, Ouija, Ozone.