Directional Communication using Spatial Soundin Human-Telepresence

Shohei Nagai∗ Shunichi Kasahara† Jun Rekimoto‡∗†‡The University of Tokyo †‡Sony Computer Science Laboratory

∗shohei.nagai14@gmail.com †kasahara@csl.sony.co.jp ‡rekimoto@acm.org

ABSTRACTCommunication is essential for working effectively with oth-ers. We communicate with each other to share their situationand what they are thinking. Especially, using voice is oneof the most common ways to communicate. In previous re-search, we proposed LiveSphere that shares the surroundingenvironment with a remote person and provides immersiveexperience to effectively collaborate with each other. Thissystem realizes ”human-telepresence” where a person can bein other person and experience the environment. However,the communication in human-telepresence has some prob-lems. In this paper, we propose directional communicationwith spatial sound to alleviate the problems. We also reporton the result of user study.

Author KeywordsSpatial sound; Remote collaboration; Omnidirectionalimage; First person view streaming; Wearable computer;Human-telepresence

ACM Classification KeywordsH.5.m. Information Interfaces and Presentation (e.g. HCI):Miscellaneous

INTRODUCTIONCommunication is essential for working effectively with oth-ers. We communicate, both verbally and non-verbally, to un-derstand each other’s situation and what they are thinking. Inremote collaboration, sharing sound and visual informationis one of the essential factors for realizing efficient collabora-tion[3].

We proposed LiveShere[4] to collaborate with a remote per-son (Figure 1). This system records the environment sur-rounding a person in the actual situation (referred as to theBody) in the form of omnidirectional images and shares themwith a remote person (referred as to the Ghost). The imagesare decoupled with ego-motion, so Body and Ghost can ob-serve the shared environment independently. This approachesrealizes ”human-telepresence” where a person can virtuallybe in other person and observe the environment as if present.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from Permissions@acm.org.AH’15, March 09 - 11, 2015, Singapore, Singapore.Copyright 2015 ACM 978-1-4503-3349-8/15/03...$15.00http://dx.doi.org/10.1145/2735711.2735818

In human-telepresence, Body does not communicate face toface with Ghost but communicates with Ghost who is virtu-ally present in themselves and independently observe the sur-rounding environment from the same point of view of Body.In the situation, there are some problems with conventionalvoice communication.

In this paper, we propose communication using spatial soundin human-telepresence. First, we discuss the existing prob-lems in voice communication in conventional LiveSpheresystem. Then, we describe the prototype of communicationsystem using spatial sound. We also report on the user studyfor evaluating the effectiveness of spatial sound and discussthe insights derived from the experiment.

Voice communication

Omnidirectional imagesRotation data

Body user Ghost userFigure 1. LiveSphere overview: Body user wears the head-worn deviceand share the environment using omnidirectional images. The captureimages are stabilized and transmitted. Ghost user observes the sharedenvironment independently from Body’s ego-motion.

COMMUNICATION PROBLEMSIn a previous study, we could found Body and Ghost indepen-dently observe and comprehend the environment using Live-Sphere. On the other hand, we also observed some problemsas shown below.

Difficulty in understanding Ghost’s situationIn conventional LiveSphere system, Body communicateswith Ghost just with ordinary voice chat, so it is difficult tounderstand what direction Ghost looks toward and what s/hefocuses on. This prevents the understanding of the partner’ssituation. Actually, in the interview after the task, some par-ticipants claim about this issues and even show the fear like”He (Ghost) is watching everything I’m doing.”

Difficulty in giving directional guidanceIn a collaborative task, they also had difficulties in giving di-rectional guidance. In the task, some participants as Ghost

159

uses directional expressions like ”Turn left.” However, suchkind of expressions is difficult to pinpoint a specific placeand direction. It is observed that Body passed through thedirection Ghost expects Body to turn toward. In a local com-munication, to indicate the specific direction, they can usegesture like pointing to a specific point, or move and speakto from the place on the direction like ”Come here.” On theother hand, both cannot be used in the conventional system.

COMMUNICATION WITH SPATIAL SOUNDHuman has an ability to localize a sound source using theirauditory system[1]. Spatial sound uses this human character-istic to allow us to virtually perceive sound from a specificplace. Using spatial sound, we can add intuitive directionalinformation to voice communication. This enables Body tounderstand which direction Ghost is looking toward whentalking, which alleviates the above-mentioned problems.

IMPLEMENTATIONWe use spatial sound in both Body and Ghost sides. We syn-thesize spatial sound using openAL. Here, spatial sound isdefined with two elements: sound source and its position. Wecapture Body voice as sound source for Ghost using head-set, and vice versa. On Body side, the position of the soundsource is defined on the point of a sphere on Ghost’s rela-tive head direction from Body’s head direction Rgb, and viceversa on Rbg (Figure 2). These are calculated from Body’shead rotation and Ghost’s head rotation from the basic di-rection, both of which are estimated in LiveSphere. At first,Rbg is calculated from the head rotation data transmitted fromBody and the captured Ghost’s head rotation on Ghost side.Then, spatial sound is immediately synthesized by OpenALusing the calculated position of sound and the sound transmit-ted from Body through the Internet, and provided to Ghostusing headset. On Body side, the calculated sound positionand the captured sound are sent from Ghost. Then, spatialsound is synthesized and provided to Body.

Ghost’s voice position

Body user Ghost user

Rgb

Rbg

Body’s voice position

Figure 2. Communication with spatial sound: The position of Ghostsvoice on Body side is defined on the point of a sphere on Ghosts relativedirection from Bodys head direction and vice versa on Ghost side.

USER STUDYWe made user study for gathering feedback and evaluat-ing how communication using spatial sound alleviating theabove-mentioned problems. Six persons participated in thestudy. In the study, Body participant is in a laboratory andGhost participant is in a different place. Here, Ghost is told

to give guidance for Body to pick up specific objects, whoseposition was taught to Ghost in advance. All participants playroles of both Body and Ghost. After the task, we interviewedthem and recorded their feedback.

RESULT AND DISCUSSIONThe result showed Body and Ghost could effectively com-municate using spatial sound system. Participants reportedthat spatial sound was useful to understand the situation ofthe partners. Some of them comments that they felt ”He wasin the same place.” On the other hand, some participants asBody claimed that it was difficult to estimate which direc-tion the Ghost look toward while s/he said nothing. Aboutthis issue, one participant comments ”Noise (included in thetransmitted sound data) was a cue to estimate the partner’sdirection when he said nothing.” The noise was also spatial-ized. This implies that, other than human voice, continuoussound directed to the partner’s direction is useful to estimatethe partner’s situation.

One participant reported that ”It would be useful to put asound marker to indicate a specific direction and object.”Human has an ability to differentiate and provide attentionto given sound in the presence of multiple sound sources[2].This suggests the possibility that Body effectively use suchsound marker to understand the partner’s indication. More-over, this can also realize that multiple Ghost users simul-taneously give guidance using spatial sound. Then, Bodydifferentiates them and selectively understands what they aresaying.

CONCLUSIONIn this paper, we propose a novel communication with spa-tial sound according to the relative head direction in human-telepresence. We developed the prototype of this methodand made user study for evaluating this method. The resultshowed using spatial sound improved the understanding ofthe partners’ situation. Conversely, the study also impliedsome assignments for the method. We discussed about thatand proposed design implications. For future work, we willimplement them and make more detailed evaluation.

REFERENCES1. Blauert, J. Spatial hearing: the psychophysics of human

sound localization. MIT press, 1997.

2. Broadbent, D. E. Perception and communication.Elsevier, 2013.

3. Fussell, S. R., Setlock, L. D., and Kraut, R. E. Effects ofhead-mounted and scene-oriented video systems onremote collaboration on physical tasks. In Proceedings ofthe SIGCHI Conference on Human Factors in ComputingSystems, CHI ’03, ACM (New York, NY, USA, 2003),513-520.

4. Nagai, S., Kasahara, S., and Rekimoto, J. Livesphere:Sharing the surrounding visual environment forimmersive experience in remote collaboration. InProceedings of the Ninth International Conference onTangible, Embedded, and Embodied Interaction, TEI 15,ACM (New York, NY, USA, 2015), 113-116.

160