Abstract «eskin» has been influenced by working with researchers in artificial intelligence, neuroscience and wearable computing who examine tactile perception, embodiment and brain plasticity. In this article I trace the inspirations, the challenges and the out-comes of this project with various scientists and technicians on the development of a prototype and the problems that evolved. What happens when the aims of projects are not clearly defined from the beginning of the project or when the funding is not adequate? It seems that experiment building in trans-disciplinary teams can be both, engaging as well as disruptively disengaging for artists and for users. This paper shows how the processes of production itself can either be encouraged by creative user en-gagement or driven by passions of artistic inquiry or stalled by disciplinary traditions and problems of communication.
Keywords Visually impaired users, art and science, wearable interfaces, tactile and sound perception.
History of the Project «eskin» began with my own “artist in residency” at the Artificial Intelligence Lab at the University of Zurich, where researchers have been studying neural networks to understand how an animal’s sensory perception and reac-tion to its environment can contribute to its autonomous behavior. The engineers and programmers there are partic-ularly concerned with bio mimicry, navigation, locomotion and embodied behavior. They develop engineering and software solutions to mimic these functional concerns. These tasks often require the trans-disciplinary collabora-tion of people from other disciplines. My own interest in our skin and its sensory perception came from my earlier project called Digital Body Automata (1995) at the Media Museum, Centre for Art and Media Technology (ZKM) in Karlsruhe, Germany [1]. In 2002, I discussed collaborating on a similar project with senior researcher Daniel Bisig at the A.I.Lab in Zurich. We originally thought that we might be able to design an embodied tactile system for autono-mous mobile robots enabling them to react with skin re-sponses akin to those of humans in foreign environments like on the surface of the moon. Daniel wanted to attempt to understand neural control, morphology and bodily sys-tems by building electronics to mimic the peripheral senso-ry nerves and motor coordination. We wanted to add the effect of using audio-visual tools on the process of
with other cortices. Daniel was interested in the communi-cation between the sound cortex and the somatic cortex and the effect of combining sound and tactile recognition. By working with Fumiya Iida, an engineer in the Zurich A.I.Lab, we used inexpensive, off-the-shelf sensors to mimic the modalities of human skin perception. This pro-ject we called «eskin». The initial aim of «eskin» was to understand the relation ship between touch and sound by developing an intelligent artificial skin. However I strongly believed that the best way to understand the modalities of skin perception or to test the potentials of cross modal interaction, was to build an interface to help augment vi-sion lost by the visually impaired, but not everyone in the team totally agreed with me.
Three Stages of Development In retrospect it was this disagreement that led to the way that the interface developed. The first stage was an immer-sive installation with a set of interactive interfaces, the second was a series of workshops for the visually impaired and the third initiative was the design of a wearable proto-type. Stage One. With the help of Andreas Schiffler a com-puter programnmer and physicist, Daniel and I built three «eskin»-shells with off-the-shelf electronics, which at-tempted to mimic the modalities of pressure temperature, vibration and proprioception. Also we built a three-screen audio-visual projection to see how the resultant interfaces might be able to trigger animations and sounds in “real time”. (Fig.2) The objects used basic wireless portable PICs or programmable micro-controller-based technology, body temperature sensors, piezoelectric vibration sensors and pressure pad sensors. The modality of proprioception was mimicked by infrared tracking technology and tilt sensors imbedded in all three interfaces. These interfaces were linked to a central Linux server and three client Mac computers running Java Scripts. Through these interfaces and clients, the users could trigger parts of three mytholo-gies about skin from Egypt, China and Australia, portrayed with texts and dancers. Each of these creation myths de-scribed how the skin of cultural characters created the landscape. Isis, the Egyptian god and Pangu, the Chinese god, were said to create the soil from their skin, while in Aboriginal dreamtime, the Thorny Lizard of legend con-structed the landscape by moving through it and pushing it into various 3-dimentional forms whilst shedding his skin pigment for its color. (Fig.3) Through the interfaces view-ers were able to manipulate the animation speed of the dancers as well as change the landscape backgrounds and the sound. At this point we went for a grant from a Swiss foundation to further our research but we were unsuccess-ful. Then we acquired some seed funding from the KTI (Ministry for Innovation and Technology), a funding body supporting Swiss industry developments in collaborations with Swiss universities. which caused a new goal orienta-tion for our project. However before we dismantled stage
the Blind) in Zurich [2]. Could these users shed light on how to develop «eskin» as a wearable device as American neuroscientist Paul Bach y Rita once suggested [3]? To-gether with Valerie Bugmann, Marille Hahne and the Ca-rambole dance theatre group Zurich, we hoped to explore orientation, cognitive mapping and external audio-visual device control, as well as to convey spatial information through pattern stimulation directly on the human arm [4]. At the outset, the workshop participants expressed their feelings of exclusion from visual culture and theatrical performances and their desire to create cultural events for sighted audiences. As the first step towards this unusual dream, we tested their response to feedback and electro-stimulation on their skin, to improve their communication potentials. We also designed and constructed a crude wear-able circuit embedded with micro-sensors, actuators and
pocketsize computers with wireless connections to sound devices. In addition, we began to design a new mediated platform, which would be customizable in response to wearable tactile and sound feedback. Could «eskin» be attached to their bodies and to other objects on the «eskin»-platform and communicate feedback to the user in the form of sound and tactile data? This question led to five differ-ent workshop activities geared toward participants of dif-ferent levels of visual acuity. Consequently, we set up as series of workshops. The first was to perform tests with Radio-Frequency Identifica-tion (RFID) readers. The aim was to discover how individ-uals associate sounds with ordinary objects and situations (like the sound of a knife cutting through bread) compared to more abstract associations (like the sound of rain drops while touching a printed Braille matrix). The second task was designed to determine how the participants would react to electro-touch sensitivity and pattern recognition using a micro-array of pins stimulating the skin on their arms. The aim here was to see if they had difficulty recog-nizing patterns with dots less than 2 cm apart, however we did find that a type of electronic Braille electro-pattern-stimulation, could be easily learnt. A third task was to test for tactile substitution by using a cognitive mapping exer-cise. Participants wore a grid on their arms and attempted to locate themselves on an associative grid on the floor. These tests proved that tactile stimulation from an interface on the arm could increase their abilities to navigate and orient themselves. In a forth workshop, dancers worked with the participants on improvisation exercises and ges-ture-based communication. What relationship do the par-ticipants have to their own bodies? How can touch help them to communicate to others? The above tasks improved their perceptual faculties, focused their attention and in-creased their motor abilities! The last task was to deter-mine whether sound transmitted through “bone-phones” transducers on the foreheads of the participants could be an aid in navigation and orientation, a factor that would leav-ing their ears free for eco-location. After stage two we were left with the realization that there are not many devic-es that work well, are cost effective, and/or allow for the personal sound cue preferences and sound customization they wanted to create. Non-verbal guidance sounds were considered preferable, because they are so clearly distin-guishable from speech. From these five workshops, we concluded that visually impaired participants would like to take control of audio-visual devices through the use of more intuitive electro-tactile responses and replace the rather inadequate HCI (Human Computer Interfaces) that currently exist for them. Stage Three: So with the advice of the Wearable Com-puter Lab at the ETHZ, we designed a device to integrate an embroidered circuit that could attach to the arm, and control a mediated stage. The result had to be flexible and portable enough to accommodate pressure, vibration, and temperature sensors and also to support a motion sensor for gesture recognition. We worked with a company to con-struct a wearable electronic circuit that could act as a sub-
strate base for these sensors. This embroidered circuit consisted of two layers designed to maximize the levels of resistance between them. Temperature feedback transmit-ted through the skin would be a proximity detector, while pressure pads produced coded messages onto the skin. This design included integration of this electronic circuit with off the shelf bone phones for sound, compasses for vibration direction, accelerometers for gesture recognition and ultrasound to avert collisions with obstacles. (Fig.5-7)
Fig.5 The wearable embroidered circuit inspired by Braille pat-terns. Designed by Jill Scott, Valerie Bugmann. Constructed by Bischof Textiles, St. Gallen
Fig.6 Stage Three of «eskin»: Mobile wearable armband inter-face, embroidered circuit, sensors and shoulder pad hooked up to the QBiQ computer from the Wearable Computer Laboratory. Swiss Federal Institute of Technology. Design Jill Scott (2008)
Analysis – Transdisciplinary Collaboration These three stages have caused related challenges and disruptive reactions. In the first stage, we did find a way to electronically mimic four modalities of the human skin perception, but the question of who this interface was be-ing made for was not clear. I seemed to be the only one who wanted to take into account the creative potentials of the visually disabled and incorporate their expertise and dreams into the design of an interface. So throughout this period, although I studied more about interactive theatre, disability, cognitive psychology, neuroscience and engineering, the need for us to be part of an even larger trans-disciplinary team became increasingly apparent. We joined up with the Wearable Computer Lab and Prof. Moria Norrie, Global Information Systems Group, ETHZ. When we finally received developing fund-ing from the (KTI) [5] to locate users for the project, we searched for industry partners. Although we all wanted to design «eskin» as a wearable interconnected device, we did not agree about who the wearers should be. The team also
discovered difficulties communicating. While I focused on the visually impaired, the developers tended to think most-ly about commercially promising uses, that was, as the basic systems of devices for local map navigation without GPS. To this aim, we conducted a feasibility study in the form of questionnaires to see if «eskin» could be used by visitors who needed a cognitive map for the Zurich Museum Night and for the Edinburgh Theatre Festival. The results of these surveys did suggest that such an interface could improve navigation and access to information. However, I felt that my research was incomplete. How could the results from stage 2 of «eskin» be fully viable without engaging the opinions of visually impaired people? Meanwhile the engi-neers who are trained in terms of milestones and work packages wanted to construct an assembly without users’ input for the design. From the outset, the engineers were interested to build a microchip, to make «eskin» light and easy to use, and to free it from its reliance on large battery storage. This aim was estimated to require an investment of $500,000, an unheard amount for a project in the arts. We were coming to the project from a different perspective, one of knowledge production rather than the lure of profit-able markets for our discoveries. We consistently came up against the same response from potential industry partners, who would not fund such a survey, and could see only a limited number of users and a small market for «eskin»! We were offered the advice, without animosity, that we might instead explore ways in which other larger groups could use such a device and that the visually impaired might instead benefit from the upgrading of existing digital devices such as mobile phones. Alas, with all these diffi-culties and communication problems we ultimately were confronted the problem of not being able to produce «eskin» for the visually impaired audience I had envi-sioned [6].
interesting to re-adapt for «eskin». In this light we would hope that alongside with the direction championed by in-teraction theorist Paul Dourish, our focus on skin will con-tinue to be on the users levels of “ubiquity, tangibility and most of all, shared awareness, intimacy and emotion”.
References [1] Calvanesitc Skin Response was used in Digital Body Autom-ata (1998) Media Museum, Center for Media Technology: ZKM Karlsruhe, http://on1.zkm.de/zkm/werke/DigitalBodyAutomata [2] The Blindenheim Zurich / Accessed 1.3.12 http://www.muehlehalde.ch/mobile/mobile.html [3] More research info about of Paul Bach y Rita can be found on http://henkvstaden.wordpress.com/tag/brianport/ Accessed 1.3.12 [4] We tested the various types of navigation devices for the blind that already exist, f.e. Sonar: http://www.batforblind.co.nz/ and Way Finder. http://dl.acm.org/citation.cfm?id=1463179 (Ac-cessed 1.03.12) [5] KTI/CTI Swiss Federal Commission for Technology and Innovation.http://www.kti.admin.ch/ (Accessed 1.03.12) [6] Exhibitions of «eskin» include: The Welcome Trust, Exhibi-tion on Skin. June 10-September 26th, 2010, Kulturama Zurich- Neuromedia. Zurich August 2011-2012 and The Winchester Science Center 2012-2013
Bibliography Bongard, J and Pfeiffer, R ( 2007) How the body shapes the way we think MIT Press Dourish,P (2001) Where the action is: the foundations of embod-ied interaction, MIT Press Sommerer, C, Jain L,C and Mignonneau, L, (2008) The Art and Science of Interface and Interaction Design, Volume 1 Studies in Computer Intelligence Design 141, Springer Ungar, S (2000) Cognitive mapping without visual experience. In Kitchin, R. & Freundschuh, S. (eds) Cognitive Mapping: Past Present and Future. London: Routledge.
Author’s Biography Dr. Jill Scott (www.jillscott.org ) is professor for Art and Science Research in the Institute Cultural Studies in the Arts, Zurich University of the Arts (ZhdK), Founder of the Artists-in-Labs Program (www.artistsinlabs.ch) and Vice Director of the Z-Node PHD program on art and science at the University of Plymouth, UK (www.z-node.net). Her recent publications include: Neuro-media: Art and Science Research with Esther Stöckli (2012), The Transdiscourse book series:Vol.1: Mediated Enviroments, (2011), Artists-in-labs: Networking in the Margins (2011) and Artists-in-labs: Processes of Inquiry (2006). All publications are with Springer Press. Her artwork spans 38 years of production about the human body, behaviour and body politics. In the last 10 years she has focused on creative media art experiments about neuro-science, ecology and sensory perception resulting in the construc-tion of interactive mediated sculptures based on studies she has conducted in collaboration with the University of Zurich.
