FromSoundArttoVirtual/Augmented/MixedReality:theexpansionofimmersion

andagencythrough3Dandinteractivesoundindigitalmedia.

JuanFelipeCorrea

AresearchpapersubmittedtotheUniversityofDublin,

inpartialfulfillmentoftherequirementsforthedegreeof

MasterofScienceInteractiveDigitalMedia

2017

ii

Declaration

IhavereadandIunderstandtheplagiarismprovisionsin

theGeneralRegulationsoftheUniversityCalendarforthe

currentyear,foundat:http://www.tcd.ie/calendar

IhavealsocompletedtheOnlineTutorialonavoiding

plagiarism‘Ready,Steady,Write’,locatedathttp://tcd-

ie.libguides.com/plagiarism/ready-steady-write

Ideclarethattheworkdescribedinthisresearchpaperis,

exceptwhereotherwisestated,entirelymyownworkand

hasnotbeensubmittedasanexerciseforadegreeatthis

oranyotheruniversity.

Signed:_______________________________

JuanFelipeCorrea

12May2017

iii

Permissiontolendand/orcopy

IagreethatTrinityCollegeDublinmaylendorcopythisresearchpaperuponrequest.

Signed:_______________________________

JuanFelipeCorrea

12May2017

iv

ACKNOWLEDGEMENTS

IwouldliketothankNatasaPaulbergforallherguidanceandsupportinthedevelopment

ofthispaper.Iwouldalsoliketothankallthelecturersinthecoursefortheircontributions

inmyexpansionoftheunderstandingofdigitalmediainpresenttimes.

v

JuanFelipeCorrea

MScInteractiveDigitalMedia

ThesisTitle:FromSoundArttoVirtual/Augmented/MixedReality:theexpansionof

immersionandagencythrough3Dandinteractivesoundindigitalmedia.

Supervisor:NatasaPaulberg

Year:2017

Abstract

Thispaperisintendedtoexploretheoverlappingconceptsofsoundartwithvirtualreality,

focusingespeciallyonimmersionandagency.Contemporaryartinthe60s,andparticularly

soundartists,engagedinareconfigurationabouttraditionalunderstandingaboutart,its

exhibitionspaces,andtherelationshipbetweenartistandaudience.Throughthis

questioning,itexploredchangesinspacethroughsoundtranslocationandspatialization,as

wellaswithinteraction.Thisinteractive,two-waycommunicativerelationbetweenartist

andaudienceenabledpeopletoshiftfromapassiveabsorbingstatetoamorecreative

activeone.Theartistandtheaudienceareco-creators;theyareinasamelevelposition.

VirtualReality,withtheuseof3Daudioandinteraction,alsoimmersepeoplein

environmentsthatbreakdichotomiesbetweenartistsandaudience,artandperson,and

placethepeoplewithinthepieceofart.ThefeelingofimmersioninVRisstrengthenwith

interaction,whichenablespeopletoalterthecontentofthevirtualworld,hencecreatinga

soliderfeelingofimmersionthroughaconvincingsenseofagency.Digitaltechnologyoffers

newpossibilitiestoengagehumansincreativeendeavorsthatnolongerdependonlocation

ormoment.Themobilityandcomputingpowerofnewtechnologieshavetakenonemore

steptothefront,theideasthatwereonceconceivedwithincontemporaryart.

ThedevelopmentofthisworkisbasedonabackgroundresearchonsoundartandVRanda

synthesisandanalysisofconcepts.Theseconceptsarethenevidencedincasestudiesof

recentvirtualrealityprojects.

vi

TABLEOFCONTENTS

ListofFigures

1. Introduction1

1.1Contextandprojectdescription.1

1.2MotivationandGoals.2

1.3ResearchQuestionandcontribution.3

1.4PaperStructure.3

2. Background/LiteratureReview

2.1Introduction4

2.2SoundArt4

2.2.1SpaceandImmersion5

2.2.2Interaction9

2.2.3Conclusion11

2.3SoundDesignandInteractiveAudio12

2.3.1Linearmediasound(Film,Television)12

2.3.2Nonlinearmediasound15

2.3.2.1GamesSound15

2.3.2.2VirtualReality17

2.3.2.3VRSoundTechnology20

2.4Discussion25

2.5Conclusion29

3. Casestudies:ImmersionandInteractioninVR29

3.1Introduction29

3.2Works30

3.2.1GhostbustersTheVoid.30

3.2.2PlayStationVRWorlds.31

3.2.3Playthings:MusicalVRPlayground34

3.2.4GnomesandGoblins.35

3.3Conclusion36

4.GeneralConclusion37

References39

vii

Abbreviations

VR-Virtualreality

AR-AugmentedReality

MR-Mixedreality

AI-ArtificialIntelligence

viii

LISTOFFIGURES

Figure1:RecycledSoundscapes(2004).6

Figure2:SoundFiledIV(1995)7

Figure3:PlaytheBuilding(2005)8

Figure4:Vicissitudes(1998)10

Figure5:Traditionalapproachvs.participativecommunication.(2005)16

Figure6:RVContinuum.(1994)18

Figure7:Sensorama. 19

Figure8:HeadMountedDisplay. 19

Figure9:PlayStationVR,OculusRift,HTCViveheadsets 20

Figure10:ITDandIDLbasedonfrequency 22

Figure11:Front/Backambiguousperceptionofsound. 22

Figure12:VRlinearandinteractive 23

Figure13:TheVoidphysicalobject 30

Figure14:TheVoidvirtualobject 30

Figure15:OceanDescent 33

Figure16:PlayStationVRWorldsMenu 34

Figure17:PlaythingsGameplay 34

Figure18:GnomesandGoblins 35

1

Chapter1

Thisfirstchapterisintendedtoexplainthecontextoftheresearcharea,whatthisproject

consistsof,andwhyitisimportant.Itwillalsodefinethescopeofthisresearch,its

motivations,goals,andthegeneralstructureofthepaper.Itwillpointoutthepresentrise

ofVirtual(VR),Augmented(AR)andMixedReality(MR)technology,andhow,throughan

analyticaldescriptionandcomparisonofsoundartandVR,aparallelcanbecreatedto

mergebothfields,exploringnewcreativepossibilities.

1.1Contextandprojectdescription

Thereisaninevitableriseinpresentdaysofvirtual,augmentedandmixedreality

technology.Afterdecadesofresearchanddevelopment,thetechnologythatwasonce

availableonlyonspecialresearchfacilitiesisnowbeingofferedmassivelyforthegeneral

public.OculusRift,HTCVive,GearVR,GoogleCardboard,MicrosoftHololensandmany

otherdevices,arenowavailableforconsumers.AccordingtoDigi-capital(2017),theAR/VR

marketin2016was$3.9billiondollars.Eventhoughthehypearoundthistechnology

createdbefore2016hasstumbledwithreality,generatinglessrevenuethanexpectedand

shiftingthefocusmoreonARthanonVR(Digi-capital2017),thefactisthatthistechnology

hasfinallyarrivedandthisisthereasonwhyitisnecessarytodiscussandresearchthe

possibilitiesthatthisnewmediumoffersindiversefieldslikearts,entertainment,

enterprise,andeducation.

Forthispaper,theareaofresearchwillcenteronspaceandinteractioninsoundartandVR,

andtheparticularroleof3Daudioandinteractivesoundinnewmedia.Thefirstsectionof

thisprojectwillintroducebasicconceptsofsoundartfocusingonspaceandinteraction.It

willthenexplaintheroleofsoundinlinearandnonlinearmediaandthroughthis,compare

conceptsandtechniquesusedinsoundartinrelationtoVR,findingsimilaritiesand

differencesbetweenbothmediums,andshowinghowtheconceptsofbothworldsoverlap,

thusprovidinganinterestingnewareaofexperimentationanddevelopment.

2

1.2MotivationandGoals

Therearethreemainmotivationsforthisresearch.Thefirstoneistocompensatethe

traditionallackofimportancegiventosoundasanareaofresearchandasamaincreative

toolinmedia.AsFraninovicandSerafin(2013.p.1)state,“soundhasbeenaneglected

medium,withdesignersrarelyawareoftheextenttowhichsoundcanchangetheoverall

userexperience”.MichelChion(1994)alsoclaimsthissituationstatingthattraditional

theoriesaroundcinemahaveeludedorsubestimatedtheroleofsoundanditsimportance,

eventhoughfilmandtelevision,aswellasothertypesofmedia,providenotonlyvisualbut

alsoauralinformation.Asecondreasonfortheresearchistheexpansionofthisnew

technologyinrecenttimes,asitcanbeseeninthefinancialearningsstatedintheprevious

section.Eventhoughtherehavebeenpreviousattemptstocommercialize,forexample,

virtualrealitytechnologiesliketheVirtualboyofNintendo,thesetechnologiesneverhad

enoughcommercialsuccess,technologicalevolutionorfinancialinvestmenttoexpandas

theydotoday.Ontheotherhand,twoyearsago,asSolomon(2014)informedinForbes

magazine,FacebookdecidedtobuyOculus,amajorVRdevelopingcompany,givingthe

sectoramassivefinancialboost.ThisabsorptionofOculusbyFacebookcombinedwiththe

impressiveexpansionofsmartphonesisalsoveryimportantfortheriseofthisnew

mediumsinceARandVRcontentcanbedevelopedforthishardware.AccordingtoThe

Economist(2015)halfofthepopulationontheplanethasasmartphoneintheirpockets,

anditwillprobablygrowto80%ofthepopulationby2020,amplifyingmassivelythe

possibilitiesofofferingAR,VRandmaybeevenMRexperiencestohugeamountsofpeople.

Thispresentcontextmakesitinevitabletofocusonthistechnologyanditspotentialuses.A

thirdmotivationfordoingthisresearchisrelatedtoartandtheconceptualchanges

promotedinthe60´sbycontemporaryartists,especiallyfocusedonsoundart,who

changedtheinstitutionalizedtraditionsaroundart.Itisinterestingtoexplorehowthese

ideasareapplicableinthenewdigitalera.AsClaudiaTittel(2009)explains,sincethe

1960´s,ashiftinthewaythatartistsperceivedaudiencesandartitselfcreatedasubversion

oftraditionalconventionsregardingparticipation,space,andimmersion.Theideasthat

motivatedtheexpansionofsubversiveartistsincludedtheruptureofthedivisionbetween

artistsandaudience,theco-creationofanartwork,andthealterationofspaceamongmany

others(Licht,2009).Itisinterestingthentoseehowtheseideasareoverlappedoncurrent

3

socialandtechnologicalcontextstoseehowtheyarestillapplicableandhowtheycanbe

expandedinwaysthatpreviousartistsneverimaginedpossible.

Onegoalofthispaperisto,first,makeanoverviewofsoundinmediaandseeits

applicationsincurrentinteractivetechnologiesfocusingonimmersionandagencyinVR.A

secondoneistodrawaparallelbetweensoundartandVRtoestablishcommonpointsand

exploreinteractivesoundasakeyelementforimmersionandagency.Thepaperaimsalso,

ingeneral,toprovideevidenceabouttheusefulnessofsoundasatoolforinteractivemedia

capableofenhancingimmersionandinteraction.

1.3ResearchandContribution

ThedigitalworldhasevolvedexponentiallytothepointofofferingtoconsumersVR,AR,

andMRdevices.Howcanartistsandaudiencesinteractinthesemediums?Whatsenseof

agencyisgiventoapersonwithinVR?Whatdifferenceexistsbetweentheimmersive

experiencescreatedinthe60´sandthesonicworldscreatedwith3DAudio?Theaimofthis

paperistodescribeimmersionandagencyinsoundartandseeitstranslationintovirtual

realitytounderstandhowthroughinteractivesoundinVR,thesetwoelementsarefurther

enhancedandexplored.Theanalysisofsoundanditsinteractiveapplicationswill

contributealsototheexpansionofprevioussoundinstallationscenariosthatwerecreated

beforethedevelopmentofdigitalmedia.Afterresearchingaboutsoundinteractivityand

soundart,thereisagapinvolvingthepossiblemergingofthesetwofieldsandageneral

needtodevelopinteractiveaudioanditspotentialbenefitsinanevergrowingdigitalworld

thatneedsexplorationandexpansioninthewayhumansrelatetodigitalcontent:HCI

(humancomputerinteraction)usingsound.

1.4PaperStructure

Thispaperconsistsoffoursections.ChapterOneistheintroductiontothesubject,which

includesthemotivationforthisresearch,aswellasthegoalsofthepaper.Thesecond

chapteranalysessoundartandspecificallypointsouttheideasofagencyandimmersion

promotedfromthisdiscipline.Anothersubsectionofthischaptergivesageneraldistinction

betweensoundforlinearandnon-linearmediaandintroducesspecifictechnical

4

characteristicsofVirtualRealityAudio,anditincludesadiscussionsection,whichexplores

possibledevelopmentspresentonlyinthedigitalworld.Thethirdchapterofthepaper

analysesdifferentVRprojectsthatevidenceinteraction,immersion,andagencymaking

comparisonswithsoundartprojects.Chapterfourprovidesconclusionsaroundthis

comparisonbetweensoundartandVRprojects.

Chapter2

2.1Introduction

Thischapterwillintroducesoundart,explainitsconceptsandoriginsanddefinetwomain

elementsthatwillbeanalyzedindetail.ThesetwoelementsareSpaceandInteractivity.

2.2SoundArt

Findinganabsolutelyprecisedefinitionforsoundartisverydifficult.AsLicht(2009)states,

differentauthorshavetriedtodefineitassoundsculptures,soundwithvisualartconcepts,

orjustartwheresoundisaprimarymaterial.Inhisarticle,hestatesthatthistypeofartis

notconstrainedtoaspecifictimeperiod,aspecificgroupofartistsorauniquelocation.This

ambiguitysupportsthevaguenessandinconsistencyofthetermbutLicht(2009)still

sustainsthattheterm‘soundart’isusefulwhendefining“siteorobject-specific”sound

worksthatarenotmusic.Forunderstandingsoundart,isusefulthentoseesome

characteristicsthatdefinetheartformbutsinceitcomprisesawiderangeofelements,this

paperwillfocusonlyonspacereconfigurationandinteraction.

Sincethebeginningofthe20thcenturyandwithparticularcasesstartingduringthe60s,

sonicandvisualartistswhowantedtochangethetraditionaldichotomybetweenartists

andaudiencepromotedahugechangeinthewayartwascreatedandabsorbed.

“Sincethenineteenthcentury,forinstance,themodernconcerthallhasdevelopeda

physicalandconceptualsegregationbetweena“performingspaceandalistening

space”(BlesserandSalter2007,130)…listenerssitinsilence,thoroughlyimmersed

andemotionallyengagedinthemusic,yetunabletoaffecttheflowofsound.The

5

concertauditorium´sdesign,ChristopherSmall(1998,26-7)argues,notonly

“discouragescommunicationamongmembersoftheaudience”,itisalsoplanned“

ontheassumptionthatamusicalperformanceisaone-waycommunication,from

composertolistenerthroughthemediumoftheperformers.“(Rogers,2014p.17)”

ArtistslikeJohnCage,MaxNeuhaus,andLaMonteYoung,amongothers,consciousofthe

rigidityofthesephysicalandculturalstructures,startedtochangethistraditional

conceptionofcommunicationandshiftedtoideaswheretheconceptsofmusic,sound,

spaceandaudiencerolewerealtered.AsLicht(2009)states,JohnCage´s4´331,forexample,

breaksthetraditionalhierarchicalrelationbetweencomposerandaudienceandinstead

createsa“listenertolistener”connectionbetweenthem.Anotherinterestingaspectof

soundartrelatestothestrongcriticismtorestrainingandinstitutionalizedspacesrelated

toartexhibitions,museums,andauditoriums.Inherarticle,Tittel(2009p.59)saysthat

“artistwereleavingtheinstitutionalframeworkofmuseumsandgalleries…asaprotest

againstconventionalartpractices”,andsheusesMaxNeuhaus´’DriveinMusic’(1967)

soundinstallationasanexamplewherethepublicspacebecomesthenew“gallery”to

“makemusicforall”.Soundartrelatessoundwitharchitecture,environments,interaction,

space,designandecologyamongotherareas.(Licht2009,Tittel2009).Sincesoundarthas

manydifferentelements,thetwomainonesthatareexposedwithmoredetailinthispaper

arespaceandinteractivity.Theseelementsaretheonesthatcanclearlybetransposedinto

virtualrealityandtounderstandthembetterthefollowingsectionwillusesoundart

projectsasawayofexplanationandunderstandingoftheseideas.

2.2.1SpaceandImmersion.

Oldenburg(2013)saysthatsoundartexplorationsincludeexperimentsarrangingsounds

inspace,theeffectsofsoundonthebodyandsonification.Toseesomeofthe

rearrangementsofspacewithinsoundartswecanstartbyusinganexampleshownby

Licht(2009);aworkofMaryanneAmachercalledCityLinks.Heresoundsaretakenfrom

urbanenvironmentsandaretranslocatedtootherspaces.InCityLinks#6,forexample,

14’33isaworkconsistingofthreemovementswherethemusiciandoesn´tplayasinglenote.Allthatisheardistheenvironmentalsoundofthemoment.(Hermes,2000)

6

MaryannecapturestheambiencefromaBostonharborandsendsitmodifiedtoagallery

space.

“City-Linksisapieceinwhichsoundingresourcesof2ormoreremotelocationsare

fedbacktoeachothertoallowforinteractionbetweenmenandsoundatdistant

locations.”(AmacherinLudlow,2010)

Inthisproject,thereisatransformationofaspacebasedontransformationsofthe

perceivedenvironments.Thesesonicvariationsoftheenvironmentsuggest

transformationsnotonlyofaperceivedsoundbutalsoofaperceivedreality.

Inanotherexampleofsoundinstallations,HansPetersKuhnreconstructedthesoundsofa

steelworkthatwasnotlongerfunctioningaswellasRonKiuviladidwiththerecreationof

thesoundsofafactorythatonceexistedinaspecificlocation(Licht,2009).Sound

installationsreestablishalocation,aspaceandthereforeareality(oratleastpartofit)with

theuseofsound.

“Soundcanhelptostructureroomsanddifferentiatebetweenfunctionsand

purposes…maybuildtheillusionofcertaincountriesorlocations.”(Herzer,2014)

Thetransformationoflocationscanbeeitherininternalorexternalpublicplaces.As

FraninovicandSalter(2013)wrote,theprojectRecycledSoundscapes(2004)wasmadeina

publicspacewhereanartifactcaptureddistantsoundsandreproducedthembackthrough

speakersbuiltinotherartifactsonthatspace(seeFigure1).

Fig.1RecycledSoundscapes(Source:Zero-Th.2004)

7

Afterbeingcaptured,thesoundbeingplayedbackcouldthenbealteredbyanypasser-by

thatfeltinterestedtointeractwiththeobject.Withurban-publicprojects,theidea,

accordingtoFraninovicandSalter(2013p.54),istochallenge“legitimizedidentitiesand

socialcodes”,expanding,evenmore,thedisruptionofcontemporaryartnotonlyagainst

traditionalartspacesbutalsoagainstconventionalsocialstructuresandbeliefs.Insound

art,then,thetreatmentsofspacerelatetotransformationsof1)publicareas,2)interior

spacesbasedonrelocationsormutationsofexternalenvironments,asseeninCity-Links,

anditalsosometimesdrawsitsinspirationfrom3)internalstructures(Licht,2009),as

exposednext.Thearchitectureofcertainlocationsaffectsthewaysoundistransmittedand

thus,thewayitisperceived.TheprojectcalledSoundFieldIV(1995),byBernhardLeitner,

usescoveredloudspeakersplacedonthefloor(seeFigure2).

Fig.2SoundFieldIV(Source:BernardLeitner1995)

Thelowsoundsarekeptonthegroundleveland,whenpeoplemovewithinthespace,these

createasensationofmovementthrough“auralwaves”(Licht,2009).Inanotherexample,

wecanseetheuseofthebuildingstructureitselfbecomingpartofthesoniccreationwith

DavidByrne´sPlayingtheBuilding(2005).Herewefindaworkusingplumbing,beams,and

pipesthatareconnectedtothebuildingaswellaswithanorganwhichcanbeplayedbythe

visitors.Theroomitselfbecomesaresonatingbox,aninstrument(seeFigure3).Herethe

useofthestructureismoredirect.Klein(2014)explainsthatsoundartistsusetheacoustic

8

characteristicsofaspace,itsarchitectureanditsvisualtraitstotransformitorreinterpret

itthroughtheuseofsound.

Fig.3PlaytheBuilding(Source:Byrne2005)

Onefinalidearelatedtospaceisimmersion.

“Immersiveenvironmentsthatremapspectatorialhabitsfromone-way

communicationtotwo-wayactivityhelptobindspectatortospectaclebyremoving

thebarriersofpassivityandthephysicalspacebetweenviewerandartexhibition;

listenerandmusicrecital.”(Rogers,2014,p.20)

Theideaofbreakingtheseparationbetweenaudienceandartworkbringsthepersoninto

theworldoftheartpiece.MaxNeuhaus(1993)saysthatwhenhecreatesspaces,the

listenerthatentersintothemslowlychangesitsperceptionofitwhenitstartsfocusingon

thesound;thescalesofthesonicelementschangefromanalmostimperceptiblestatetoa

veryconsciousperceptionoftheauralenvironment.Hisworksimplyabodilyimmersionof

theparticipants.Rogers(2014,p.21)alsotalksaboutthisideawithBillViola´sworkslike

FiveAngelsoftheMillennium(2001)andOceanwithoutaShore(2007)wherean

audiovisualimmersioniscreatedtodetachthepersonfromtherealworldand“theresult

isakintojumpingintoapaintingorintothediegesisofafilm”.

Whentalkingaboutimmersionincontemporaryart,itispertinentalsototalkaboutsound

spatialization.Theruptureofthedivisionofthestageandtheaudiencewasexploredby

movingthesoundingsourcefromthecentralstageintothesurroundingsoftheaudience.

9

Licht(2009,p.4)statesthatduringtheseventeenthandnineteenthcenturies,thehallswere

developedtopropagatesoundcomingspecificallyfromthestage.WorkslikeAntiphonyI

(1953),wherefiveorchestraswereplacedaroundanauditorium,orGesangderJünglinge

(1956),fromStockhausen,whichusedfivespeakersaroundahalltoplaybacksounds,

changedthecentralizedsourcesofsoundandmovedthemtooccupythespaceand

immersetheaudiencesinthesonicenvironment.Infollowingsectionsregardingfilmsound

andtheperceptionofsound,we´llseehowimmersionthroughsoundspatializationwas

createdforfilmandhowitevolvedto3DaudioinVirtualReality.

Inconclusion,wecanseethreecharacteristicsaboutspaceinsoundart.Oneinvolvesthe

useofexternalenvironmentsorsyntheticonesthatcanbemodifiedandtranslocatedto

changeanotherplace´sambience,eitherapublicopenspaceoraninteriorone.Thesecond

characteristictakesintoaccountthestructureofaninternalspace,itsarchitecture,textures,

andgeneralorganization,usingthemtoalterthewaysoundispropagatedandthusmodify

howitisperceived.Thethirdelementisrelatedtoimmersion,andhowtheexpansionof

thesonicambienceandtheincorporationoftheparticipantintotheartproject,dissolves

thedivisionbetweenrealityandthecreatedworldandalsobreaksthetraditionaldivision

betweenartistsandaudience.Thenextsectionwillnowtalkaboutinteractioninsound

arts.

2.2.2Interaction.

Asseeninprevioussegments,soundartisstronglyinvolvedwithspatialreconfigurations

butalsowithinteraction.Itsupportedtheideaofbreakingthewallthatdividedtheartists

fromtheaudience,disruptingitspassivepositionandshiftingittoanactiveone.Itis

interestingtonote,though,thattheimmersiveenvironmentsdescribedbeforecould

involveparticipantsthatwerepassiveand/oractive.Thefactthattheaudienceisimmersed

inanartworkdoesn´tnecessarilymeanthatitcanchangethe“structure,contentorflowof

awork(Rogers,2014,p.21).”Thedifferencebetweenimmersionandinteractionliesin

thispart:thepossibilityofactionofthespectator.

Toevidenceinteractioninsoundart,wecanstartwithanexampleshownbyClaudiaTittel

(2009)wheresheexplainshowChristinaKubisch´sElectricalWalk(2004-2015)promotes

10

asonificationofurbanenvironmentswhereelectromagneticfieldsarepresent.Participants

ofthisprojectmovefreelyaroundthecitieswearingheadphonesthatperceivethese

varyingelectromagneticfieldsandcreatesounddependingonthesevariations.Tittel

(2009)says,“participantsarethusfreetocreatetheirownelectro-acousticalpieces.”This

projectclearlyshowsacharacteristicpresentinsoundartwheretheactionsoftheaudience

aretakenintoaccountandaffectthepieceofart.Actually,thewordaudienceinthiscontext

needstochangecompletelysinceitisnotonlylisteningtotheworkbutitisalsocreatingit;

thewordparticipantorco-creatorcouldbemoreappropriate.DriveinMusic(1967),by

Neuhaus,isanothergoodexampleofaworkwheretheinputoftheparticipantschangesthe

contentoftheartpiece.Inthisproject,agroupoftransmittersweresetontreesalonga

roadtogeneratedifferentsoundsthatvariedaccordingtoweatherconditions.The

transmittersincludedantennasthatwerepositionedtooccupyaspecificareawherecars,

tuningthesamefrequencyofthetransmitters,wouldcapturethesignal.Thefinalsonic

resultoftheprojectherewouldchangedependingontheweatherbutalsoonthespeedand

directionofthecars(Tittel,2009,p.59).Athirdexampleoftheparticipationofpeoplein

artprojectsisshownbyLicht(2009)withtheprojectcalledD.A.M.A.G.E.which,even

thoughitfallsmoreintothecategoryofexperimentalmusic,ithaselementsofinteraction

thatapplytothiscase.D.A.M.A.G.Eincorporatesaplasticbeachballasacontrollerthat

generateselectronicmusicastheballisbouncedandtranslatedaroundtheparticipants.In

thiscase,theauralresultisonlypresentiftheaudiencedirectlyactsinthepieceofart.The

projectinthiscaseisco-createdbytheartistandtheparticipantsandbreaksthe

hierarchicalseparationbetweenthem.Onefinalexampleusefultoshowinteractionwithin

soundartsisVicissitudes(1998)byCamilleUtterback(seeFigure4).

Fig4.VicissitudesbyCamilleUtterback.(Source:ElectronicBookReview)

11

Inthisproject,therearetwosoundtracks:onehasinterviewswithpeopletalkingabout

positiveaspectsoflifeormomentswheretheyfelt‘up’andanotheronewithinterviews

aboutnegativemoments,orwhenpeoplefelt‘down’.Whenpeoplegouptheladder,the

volumeofthepositiveinterviewsgoesup,andwhenapersonsituatesitselfinsidethe

outlineonthefloor,thenegativeinterviewstartstobecomeaudible.Inthissoundwork,the

personinvolvesitscompletebodytobepartofthepieceandtogeneratecontentshowing

the“embodiednessoflanguageitself.”(Utterback,2004)

2.2.3Conclusion

Interactivitywithinsoundartsrelatestothecapabilityoftheuserorparticipanttoaffect

thepieceofart.Contrastingwithtraditionaldivisionsbetweenaudienceandartist,the

destructionofthebarrierbetweenthesetwoentitiespromotedbycontemporaryartinvites

theaudiencenotonlytobeimmersedintotheartpiecebutalsotoaffectitorsometimes

eventocreateit.Theaudiencestopsbeingapassivebodyandtransformsitselfintoan

activeentityflatteningtheprevioushierarchicalstatusoftheartists.Therelationship

becomesmoreoneofa‘listener’to‘listeneror‘player’to‘player’one,asLicht(2009)

describes,thanaone-waytraditionalcommunicativesystem.

SpaceandInteractionarethetwomaincharacteristicspresentinsoundartthatcanbe

translatedintothevirtualworld.Soundartinthe60susedmulti-speakersystemsand

analogtransducerstocreateimmersionandinteraction,butthetechnologicaldigital

evolutionofrecenttimeshaveexpandedthepossibilitiesofexploringthesetwoideaseven

further.Thenextsectionwillnowintroducevirtualrealityaudiobystartingwithsoundin

linearmediaandtheswitchingtononlinear.Thestudyofsoundelementsintraditional

mediawillhelptodefinewhatauralcomponentsarepresentinmediaingeneral,whatrole

theyplayandtheevolutionofsurroundsound.Thedescriptionandunderstandingofthem

willthengiveenoughbackgroundtoanalyzeVRprojectsandoverlapsoundartsconcepts

withVRworlds.

12

2.3 SoundDesignandInteractiveaudio.

Thissectionwillgiveadescriptionofsoundbothonlinearandnon-linearmedia.Itwill

specificallyfocusonfilm/TVsoundandonvideogamesandVRsound.Thissection,

combinedwiththepreviousconceptsofsoundart,willgivethebasicconceptstoanalyze

thecasestudiesofsectionthree,thusenablingtheconnectionofbothfields.

2.3.1Linearmediasound

Tomakeadistinctionbetweenlinearandnon-linearmedia,itisusefultotakeKaren

Collins´metaphorwhenexplainingmusiccompositionsforbothmediums.

AccordingtoCollins(2007),whencomposingmusicforlinearmedia,musiciansnowhow

somethingwillsoundfromthestartpointtotheend.Thereisasequentialsetofeventsthat

followaspecifictrack:“Likebeinglockedtoatraintrack,ithasbeencomposedtostarton

onepointandprogresstoanother”(Collins2007).Onthecontrary,innon-linearmedia,

therearenopredictionsaboutwhatisgoingtohappen.Whatweencounterareoptions,

whichinsteadofbeinglikeafixedtraintrack,workmorelikeametrowhereanypersoncan

hoponandofffromanystationandchangedirectioninanywaydesired(Collins2007).

Withinlinearmedia,then,wecansituatefilmandtelevision.Inthesemediums,thecontent,

bothvisualandaural,ispredefinedandcan´tbealteredbythespectator.Itisusefultofirst

exploresoundonfilmsincemostofitsmainelementsarealsousedininteractivemediaand

theevolutionofimmersivesurroundsoundisconnectedbothtosoundartandvirtual

reality.ThespecificdifferenceswithVRregardingtechnology,implementation,

reproductionandotheraspects,willbeexplainedinthenon-linearsection.

Soundonfilmhasexistedforapproximately90years.Sincetheintroductionoflivepre-

recordedsoundplayedbackondiskin1926withDonJuanandthefollowingintroduction

ofsoundwithinthefilmitselfin1927inTheJazzSinger(Bordwell,Thompson2012),films

havebeentransformedintoaudiovisualcreationswheresoundhasbecomeamajor

elementintheconstructionofthisartform.Eventhoughmusicwasbeingplayedlivein

synchronizationwithfilmsbeforeTheJazzSinger,itwasn´tuntilsonicelementswere

recordedonthefilmitselfthattheuseofsoundasanindividual,malleable,andseparate

processopenedaworldofcreativityandpossibilitiesthatwerenotachievableinlive

13

situations.AccordingtoBordwellandThompson(2012),soundscanbedividedintothree

groups:speech(dialogue),musicandsoundeffects.Atthesametime,wecandividethe

soundfromafilmintotwosourcegroups:diegetic,whichmeansthatthesourceofthe

soundispartoftheworldofthefilm,andnon-diegetic,whichareelementsthatarenotin

thecharacters’reality.Thelinkofsoundandimageshasevolvedintoacomplexmixture

thatcombinesmusic,dialogue,andsoundeffectstocreateanillusion,an“audiovisual

contract”asstatedbyMichelChion(1994).Whatisinterestingtoseeisthatthesesound

sourcesandtypes,whichhavebeencategorizedandanalyzedintraditionalmedia,are

completelytransposabletononlinearmedia.Speech,soundeffects,andmusicarepresentin

videogamesandVRworldsandtheseareelementsthatareusedtoengageinaninteractive

waywithauserwithinavirtualworld.Sinceoneoftheaimsofthispaperistoevidencethe

usefulnessofsoundinmedia,thefollowingsectiondescribesthecurrentusesofsoundin

traditionallinearmedia,whichareingeneralalsoapplicabletoVR.Asecondreason

supportingtheinclusionofthefollowingsectionisthatinteraction,asakeyelementthat

unitessoundartandVR,isstronglyachievedthroughsonicinteraction,asitwillbe

describedintheVRsectionandasithasbeenevidenceintheprevioussection.The

possibilitiesofinteractingwiththeenvironment,virtualobjectsandevenmusicalobjects

withinVRgiveenoughsupporttounderstandwithmoredetailthefunctionsofsoundand

whattheycancreateforspectatorsandplayers.

Dependingontheintentionsofthedesignersoftheworlds,thedifferentcombinationsof

soundtypeswillcreatedifferentresults.Tohaveageneralideaaboutthecreative

possibilitiesofferedbythedifferentgroupsofsounds,we´llseethefunctionsofmusic,

dialogueandsoundeffectsinlinearmedia.Sonnenschein(2001p.155)givesfourmain

functionsformusic:

1) Emotionalsignifier:whichrelatestothesupportingroleofmusicwith“spirituality”

andemotionsofthecharactersandstoryofthefilm.

2) Continuity:musicworksasa“fill”thatcompensateslackofcontinuityeitheron

sounddialogue,ambienceoronvisualmaterial.

3) Narrativecueing:themusichelpstoestablishaparticular“pointofview”or

predefineoranticipate“athreatorasetupforajoke”,establishingstorydriven

events.

14

4) Narrativeunity:musiccangiveaformalunitybytheuseofrepetitionorvariationof

melodies,instrumentationorharmonicstructure.Theideaofleitmotivcouldbe

includedhere,whereaspecificmusicalthemeisattachedtoaparticularcharacter

orevent.

Sonnenschein(2009,p.56)citesMichelChionwhendefiningthreetypesofdialoguein

film:

1) Theatricalspeech:Thisreferstothein-storytextmadebycharacters.

2) Textualspeech:itreferstovoiceovers,whicharenon-diegeticandareusuallyfor

specialcharactersornarrators.

3) Emanationspeech:Thistextcan´tbeclearlydefinedandisusedmoretoconveya

character’spresencethantounderstandthemeaningofthesounds.

Forsoundeffects,Sonnenschein(2009)pointsouttheuseofambiencessounds,for

example,aselementsthatestablishcontinuitybetweenfilmeditsandsetsthelocationofa

scene.Foleyandsoundeffectshelptogive“arealisticsonicpresenceoftheaction”

(Sonnenschein,2009,p.40).

Besideshavingthethreemajorgroupsofsonicelementsthatarepresentinfilmsoundwe

alsoneedtothinkabouthowtheyarecombinedtoworkasaunityforthefilmandhow

theyareplayedbacktotheaudience.Theprocessofcombiningsoundiscalledmixingand

thewayitisplayedbackintheaterssetsonecommonareawithsoundartsandVR:

immersionthroughsound.Surroundsoundhasalongstorystartinginthe1940swiththe

useofquadraphonicsound,whichrevolutionizedtheuseofspaceintheaters,butitwas

onlyuntilthe70sthatsurroundsounddevelopedbyDolbystartedtobeusedmassively.

Theexpansionfromstereosoundto5.1,7.1andbeyondgavetheaudiencesabetterfeeling

ofimmersionthanprevioussettingsbuttheywerelimitedonprovidingvertical(aboveand

underthebody)sonicinformation.DolbyAtmosaswellasAuro3Darerecentevolutionsof

cinemaaudiothatcompensatethisandcreateamorecomplete3Dsonicexperience.

(Schoenherr2005;Dolby2017;Auro3D2017).

15

Soundinfilmhelpstocreatetheillusionofaworldwithsoundeffectsandasurround

environment.Theimmersionoftheaudienceintonewrealitiesispresentbothonfilmand

soundartbutinteractivityisnotpresentonfilms.Itcanbesaidthatbecauseofthis,sound

artimmersestheparticipantsintounrealscenariosinamoreprofoundwayprecisely

becauseitgivesthemthepossibilityofaction,hencebringingnearerthevirtualworldwith

therealone.Interactionisakeyelementforimmersionsothenextsectionwillnow

introducesoundininteractivemediafocusingongamesandVR.

2.3.2Non-linearmediasound

Sonnenschein(2001,p.218)saysthat“themostprofounddifferencebetweenfilmandthe

newmediaisinteractivity…theuserhasboththediscoveryanddecisiontorepeatorgo

forwardalongavariedofpaths.”SimilartoCollins´(2007)metaphorofthemetro,

Sonnenscheinshowshowoptionsandvarietyofpathsdefineinteraction.

Forthisproject,thefocusofnonlinearmediawillbeonvideogamesandVRsound.

2.3.2.1GamesSound

AccordingtoCollins(2007)soundinvideogamescanbeinteractivewhenitisresponding

todirectactionsofaplayer,anditisadaptivewhenitreactstoeventsofthegameplayor

gameenvironmentitself.ThetermdynamicaudioisusedbyCollinstodescribeaudiothatis

eitherinteractiveoradaptivedependingonwhathappensinthegame.Collins(2007)also

statesthatgamesounds,asinfilm,canalsobeclassifiedindiegeticandnon-diegeticsounds

buttheseconceptsalsoshouldincludeotherdivisionslikedynamicandnon-dynamic

soundsaswellasthelevelof“dynamicactivity”regardingtheworldofthegameandthe

player.

Aswecansee,thelevelofcomplexityregardingtheunderstandingandconceptualization

ofsoundincreasesasthepositionofpreviouspassiveaudiencespresentinfilmare

transformedintocreative,activeusers/playersininteractivemedia.Thiselementisvery

importantasitdrawsaconnectionbetweensoundartandinteractivedigitalmedia.The

linkcanbeestablishedbycomparingCollins´(2007)explanationaboutcommunication,as

seeninthefollowinggraphic,andthedescriptionsmadebyTittelaroundsoundart.

16

Fig.5.Traditionalapproachvs.participativeandnon-linearapproachtocommunication.

(Source:Collins.Essaysonsoundandvision.2007)

ThischangeofpassivespectatorstoparticipatoryusersisthesameseeninKubisch´s

ElectricalWalkssoundartprojectdescribedbyTittel(2009)whereshestates“participants

arethusfreetocreatetheirownelectro-acousticalpieces.”Aswellasalltheexamples

presentedintheinteractivesectionofsoundart,theplayersinvideogamesareactively

participatingandaffectingthecontent,theyhaveasenseofagency.AtauTanaka(2006)

saysthatagencyis“anabilitytotakeactions,tohaveinitiative.”Asenseofagencyisgiven

totheparticipantsingames,givingthemthepowertodeterminewhatandhowthings

happenwithinapredefinedvirtualworld.Similartosoundart,theartistservesasacreator

ofaplatformfromwhichtheusercreatesitsownexperience,itsownworldandtherefore

becomesaco-creatorofthepieceofart.Traditionaldesktop-basedgamesimmerseplayers

invirtualworldsthroughsurroundsound(somegamesuseonlystereosoundbutmajor

consolesincorporate5.1and7.1sound)andinteractionbutkeepthevisualaspectseparate

fromtheplayerthroughtheuseofamonitor.Infilm,asimilarsituationhappenswiththe

useofsurroundsoundbutmonitorbasedvisualinformation.

Thefollowingsectionwillintroducevirtualrealitytoseehowinteractivityandspaceare

understoodinthisenvironmentandhowimmersionchangeswithinthisrealmthatinclude

a3Dvisualfield.

17

2.3.2.2VirtualReality

Whenthinkingaboutvideogamesandfilmsitisinterestingtoseethatbothmediums

suggestanexistenceofa“virtualworld”whereplayersareimmersed,eventhoughoneis

interactiveandtheotherislinear.AsSonnenschein(2001,p.159)states,films,withtheuse

ofsound,can“inducethesenseofspace,volumeandtexture…withtheadditionofvisual

informationwecanbeimmersedinamorecompletevirtualreality.”Strictlyspeaking,it

couldalsobesuggestedthatevenpaintingsandbooksarealsoexamplesof“virtualreality”

sincetheycreatesyntheticenvironmentsintowhichviewersandreadersaretransported.

SincetheconceptofVRseemstoapplytomanyvariedmediums,itisnecessarytotryto

defineaspecificuseofthetermforthispaper.

AsMazurykandGervautz(1996,p.5)state,indifferenttypesofmediawecanfindthree

modesofimmersion:

1. DesktopVR:whichusesatraditionalmonitorfortheimage(onlyoneimage:

monoscopic).

2. FishTankVR:whichusesalsoamonitorbutincludesheadtrackingandastereo

imageofa3dspace.

3. Immersivesystems:thesesystemsusestereoscopicimagesandheadmounted

devicesinadditiontopossibleaddedenhancementsforsensoryresponselike3d

audioorhapticfeedback.

IfweconsidervideogameslikePONG(1972)orUncharted4(2016),thedifferences

betweenthemregardinggraphicandsonicquality,andgeneralgameplaycomplexityare

huge,anditcouldbesaidthat,becauseoftherealismofthegraphicsandphysicalmodeling

ofUncharted4,amoreimmersiveexperienceiscreatedfortheplayersthanwiththeolder

game.Theproblemhereisthatthereisstillacommonelementinbothgamesanditisthat

theybothremaininsideamonitoranddonotimmersetheplayerinacomplete3Dvirtual-

visualenvironment:theyarebothexamplesofdesktopVR.Asimilardescriptioncouldbe

appliedtofilmsince,althoughitcreatesanimmersivesonicenvironmentthroughtheuseof

5.1or7.1mixing;thevisualexperienceisstilllimitedtoamonitor.Thesevariedlevelsof

immersionofvirtualworldsondifferentmediamightgenerateambiguitywhentryingto

18

definewhatVRreallyis.Forthispaper,VRisreferredtothe“Immersivesystems”available,

forexample,withcurrenttechnologieslikeHTCViveorOculusRift.Theparticipantusesa

headsetthattracksmovementanditprovidesa3dvisualandsonicenvironmentthat

transportstheuserintoacompletelyvirtualworld.Wecouldusethisdefinitiontoclarify

morewhatVRis:

“Theillusionofparticipationinasyntheticenvironmentratherthanexternal

observationofsuchanenvironment.VRreliesonathree-dimensional,stereoscopic

head-trackerdisplays,hand/bodytrackingandbinauralsound.VRisanimmersive,

multi-sensoryexperience.”(Gigante,1993,p.3)

TobetterunderstandVRitisusefulalsotoseehowthisconceptrelatestoaugmentedand

mixedreality,whichareimportanttechnologiesalsobeingdevelopedinthismoment.

AccordingtoMilgram,Takemura,UtsumiandKishino(1995),allofthesevariedrealitieslie

withinavirtualrealitycontinuum.Thiscontinuumisaconceptwhereononesidewehave

totalrealityandontheotherwehavetotalvirtuality.Themiddleregionofthesetwosides

iscalledthemixedrealitywhereavarietyofcombinationsofvirtualandrealelementsare

usedindifferentdegrees.

Fig.6RVContinuum.(Source:AugmentedReality:Aclassofdisplaysonthereality-

virtualitycontinuum1994)

VirtualRealityenvironmentscreatedbytheHTCViveandOculuslieontherightsideofthe

continuum.Theworldwheretheuserisimmersediscompletelysyntheticcoveringan

entire3Dspace.Tounderstandwithmoredetailhowthiseffectisachieved,wewillmakea

generaloverviewofthehistoryofVRandthetechnologycurrentlybeingusedbythemain

VRdevelopers,focusingmainlyonsoundtechnology.Theunderstandingofthefollowing

19

sectionwillclarifyhowspaceisunderstoodinVR,thereforecreatingabackgroundusefulto

comparetosoundartconcepts.

ThebeginningsofVRcanbetracedbacktoMortonHeiligin1962whenhisinventionthe

Sensoramawaspatented.Thismulti-sensorymachineincludedastereocoloredfilm,

binauralsound,wind,scentandvibrations.(Gigante,1993)

Fig.7Sensorama(Source:MortonHeilig)

Afterthisdevelopment,IvanSutherlandconceptualized“TheUltimateDisplay”andthen

developedthefirstheadmountedAR/VRdisplay,whichincludedheadtrackingandvisual

contentthatfollowedthehead´smovement.

Fig.8Headmounteddisplay(akaTheSwordofDamocles)(Source:ForensicVR)

Aftertheseinitialdevelopments,NASAandmilitaryinstitutionsmademoreincursionsin

VRtechnologybutitwasonlyuntil1985whenVPLcommercializedtheDataglovefollowed

in1988bytheEyephone(Gigante,1993).

TheunsuccessfulcommercialresultsofVRhadtowaituntil2012whenOculus,astartup

company,managedtoraise$2.4milliondollarsandstarteddevelopingthenewgeneration

20

ofVRheadsets(C.S.W2013).Asstatedbefore,OculuswasboughtbyFacebookin2014and

onpresenttimes,VRdevelopmenthasincreasedmassivelyofferingcommerciallyVR

headsetswhichincludetheHTCVive,Oculus,PlayStationVR,SamsungGearVR,Google

Daydream,CardboardandmixedrealitydevicesliketheMicrosoftHololens(Lamkin,2017).

Fig.9PlayStationVR,OculusRift,HTCViveheadsets(Source:Wareable2017)

Tomakepossibleasynthetic3Dworld,itisnecessarytounderstandhowhumanbeings

sensetheworld.Forthispaper,wewillfocusonthewaysoundisperceivedbyhumansand

howVRtechnologyusesthisinformationtorecreateavirtualworld.

2.3.2.3VRSoundTechnology

Localizationreferstothewayhumanbeingslocatesoundsina3Denvironment,while

spatializationreferstothepositioningofsoundsina3Dspacebasedonthewayhumans

perceivesound(Hook2014).Thewaysoundisperceivedcanbesummarizedwiththe

followinglist(Oculus2017;Hook2014):

1. LateralLocalization:thisreferstosoundperceivedonthesidesofthehumanbody.

Whenthesoundhitstheleftside,thesoundreachestheleftearfirstandthenthe

rightear.Thereisadifferenceintimeandlevelbetweenthesoundsperceivedin

bothears.ForthesemeasurementstheITD(InterauralTimedifference)andtheILD

(Interauralleveldifference)areusedtocalculatethevariations.Thesechangesare

dependentonfrequencyasseenonFigure10.

21

2. Front/Back:forperceivinginformationfromfrontandbackweneedmorethan

levelandtimedifferences.Asseeninfigure11,soundscouldbeperceivedascoming

fromtheexactsamepositioniftheyareemittedatthesamedistanceandtimefrom

frontandback.Tofixthis,ourbraingetsinformationalsobythespectral(frequency

content)characteristicsofthesoundthatarechangedbytheshapeofourears,

head,andtorso.

3. Pinnae:Asseenbefore,thedifferentshapesofthepinnae(externalear)changesthe

frequencycontentofthesoundandthereforeaffectsthewaythebrainperceivesthe

soundlocation.Thesoundfromthefrontwill“resonate”withtheshapesoftheear

whilethebackwill“shadow”thesounds.

4. Head/shoulder/Torso/Elevation:Tocuethesoundsources,ourbrainuses

informationnotonlyfromdelaysbetweenearandleveldifferencesbutitalso

includestheshoulders,headshape,andtorso.Thedifferentreflectionsandchanges

ofsoundcreatedbytheshapeofthebody,givesextracluestothebraintoinfer

wherethesoundcomesfrom.Whentryingtoperceivesoundfromaboveorbelow,

theeffectofthebodyisveryimportantsinceitinterfereswiththedirectsound,

givingcluesaboutthelocationofthesonicsource.Themeasurementsthatinclude

bodyeffectsarecalledHTRF(HeadRelatedTransferFunctions).

5. Distanceandambience:TheHTRF,ITL,andIDLgiveenoughinformationtolocatea

soundina3Denvironmentbutitisnotenoughtonowthedistancetothesonic

objects.Forachievingthis,itisnecessarytotakeintoaccountloudness,initialtime

delay,theratioofdirectsoundandreverb,motionparallaxandhighfrequency

attenuation.Thefirstelementisthemostobviousone,anditshowsthatthecloser

anobjectis,thelouderitsounds.Thesecondelementreferstothetimeittakesfor

thefirstreflectiontoreachthepersonafterthedirectsound.Thethirdelement

referstothemixofdirectsoundandreverberation:theclosertheobjectis,themore

directsoundweperceiveandviceversa.Motionparallaxreferstotheperceptionof

thespeedofmovementofsoundinrelationtodistance:thefasterasonicelement

moves,thecloseritis.Finally,highfrequencyattenuationisfasterthanlow

frequency,butthischaracteristicappliesonlyforverylongdistances.Whentaking

22

intoaccountthereflectionsofsound,whatwearetalkingaboutingeneralisthe

ambienceinwhichthesoundexists.ItisthecombinationofHTRFsandreflection

detailsthatcreatethewholeinformationneededtopreciselylocatesoundina3D

perspective.

Fig.10ITDandIDLbasedonfrequency(Source:Hook.YouTube2014)

Fig.11Front/Backambiguousperceptionofsound.(Source:Oculus2017)

Onceweknowhowhumansperceive3Dsound,itistimetoactuallyimplementthis

informationtospatializeanysoundwewant.Todothis,theHTRFsareappliedtothe

sounds.SincetheaimofthispaperisnotadeeptechnicalunderstandingofVRaudio

beyondofwhathasbeenexposed,itisusefultousewhatOculus(2107)statesontheir

website,“whatmattersisthehighlevelconcept:wearefilteringanaudiosignaltomakeit

soundlikeit´scomingfromaspecificdirection.”

Asitcanbeseen,thelevelofcomplexityinwhichhumanbeingsgatherinformationto

interpretthe3Dsoniclandscapeishugesinceitinvolvesnotonlytheearsandsoundwaves

butalsothewholebody,itsshape,anditsmovementsthatsummedwithlevelandtime

differences,giveallthenecessarycuesforthebraintolocatesound.Thereasonwhythe

previousdetaileddescriptionisimportantisbecauseVRtookafurtherstepin

23

understandingwhatperceptionand,thus,immersionisandhowtocreateitinacloserway

torealitycomparedtopreviousattemptsinsoundartorfilm.Throughtheunderstandingof

thebody´sinfluenceinthesoundscapeandonperception,thetechnologygeneratesa

realisticdiscernmentofspace.

Theevolutionofimmersivesoundisastounding.Itcanbetracedbacktothequadraphonic

developmentsinfilmanditsfollowingexpansionsto5.1,7.1andAtmostechnology,ending

uptotherecent3DaudiodevelopmentsinVR.AsinfilmandVR,soundartalsoused

spatializationofsound,shiftingfromthetraditionalstage-centricsoundsourcestoamore

openfielduseofsonicprojections.Inthesethreeartfields,thereisadisintegrationofthe

divisionbetweentheaudienceandtheworkasthe“stage”isexpandedandsurroundsthe

listener.

Butonethingistocreateasurroundsonicenvironmentforausertobeinandanother

thingisinteraction:thecapacityofhumanstochangethecontentofthesonicand/orvisual

landscape.InVRthereisthepossibilityofhavingbothinteractiveandlinearcontentasseen

inthefollowingdiagrambyAltman,Krauss,Susal,Tsingo(2016).

Fig.12VRlinearandinteractive.(Source:AESImmersiveAudioforVR2016)

Thismeansthatwithinasurroundvisualandaurallandscape,itispossibletoengagein

morecontemplativeorpassiveexperiencesbutalsotoparticipateininteractionsdepending

onthecontentbeingpresented.Ifweweretoincludesoundartswithinthegraphabove,it

couldbeplacedonbothlinearandinteractivesections.Itispossibletocreatesonic

environmentswheretheusercanonlychangeitspointofviewbutnotthecontent

presented.ApassiveVRexperience,eventhoughitgeneratesanimmersivefeelingthrough

3Denvironments,lacksadeepersensationofpresenceduetothelackofinteraction.

24

Soundinteractioninvolvesasonicreactioncreatedbytheinputoftheuser.Ingames,

whetheronmonitor-basedorVRexperiences,theaudioisusuallymanagedingameengines

withaudiomiddlewarelikeWWiseorFMOD(orwithinternalsoundtoolsofgameengines

likeUnity),andtheaudioisrenderedinrealtimetogenerateanimmediateresponsetothe

player´sactions,whileforlinearcontenttheaudioispre-producedand“encoded,packaged

andstreamedtotheviewer”(Altman,Krauss,Susal,Tsing.2016,p.2).Interactioninvirtual

reality,aswellasinsomesoundartprojects,canbeexpandedbeyondthelimitedactions

basedoncontrollersorkeyboardinputstriggeringreactions,andinstead,involveamore

corporealexperience.Theinclusionofthebodyasanintegralpartoftheimmersiveand

interactiveexperienceofvirtualrealityandsoundartiswhatcreatesadeepersenseof

presenceandagency.

“Thematerialityofsoundgainsatrulyembodiedqualitythroughphysical

interaction-awayfromequatingthemediumtothephysicalsupportonwhichitis

storedandtowardthemediumofaction-soundrelationsshapingtangibleand

corporealexperience.”(FraninovicandSlater.2013.p.72)

ThedifferencebetweenVRandsoundartisthatthelevelofdetailusedtoincorporatethe

bodyasakeyelementininteractionindigitalmediumsiswaymorecomplexandrich,

thankstotechnologicaladvancements,allowingforgreaterandmorerealisticinteractions.

Thefactthat3DaudiousesHRTFsforcreatingthesoundspatialization,meansthatthe

presenceofthebodyinsideamediumistakenintoaccount.Itsmovementsandshape

directlyaffectthesoundanditsperception.Besidesthis,naturalgesturesofthebody,that

inreallifegenerateaspecificeffect,canbereproducedinVRwithmotionsensorsor3D

scannersthatperceivethebodyandincludesitsmovementswithinthevirtualreality.A

deeperexpansionofgestureandinteractionwillbediscussedinthefollowingsection.

Thenextsegmentwilldiscusstheconnectionsofboth,soundartandVR,andseehownew

conceptsandpossibilitiesarisefromthemergingofbothdomains.

25

2.4 GeneralDiscussion

Thissectionwilltakeintoaccounttheideasofinteraction(agency)andspace(immersion)

presentbothinsoundartandVR,asithasbeenshowninprevioussections,andexplorethe

connectionsandcreativeexpansionsthatcouldbemadewithinthenewdigitalmedia

world.

Soundartandvirtualrealityhaveelementsincommon.Theybothimmerseusersincreated

worldsthroughspatialconfigurationsandtheybothcangivethepossibilityofaction;give

themasenseofagencythroughinteraction.Sonicartistsfromthe60screatedworksthat

changedthewaypeopleperceivedartbyreplacingthelocationsofexhibitions,theideasof

whatanartpiecewasandalsobygivingtheaudiencethepossibilityofshiftingfroma

passivereceptivestancetoamoreactiveone.Theempowermentoftheaudience

determinednotonlytheplaceswhereartwasbeingmadebutalsoaredefinitionofart

itself,artistsandsocialconcepts.Theevolutionofdigitalmedia,aswell,hasimplied

changesthatshiftfromcontentbeingcreatedbyfewentitiestocontentbeingcreatedby

anyonefromanywhere.Fromtheexpansionofblogstosocialmedia,onlinevideos,digital

newspapers,videogames,andVR,thedigitalrealmisnurturedconstantlybyanever-

growinggroupofparticipantsthatfindinthisspaceanarenawhereexpressionisalmost

limitless.Contemporaryartistsfromthe60spre-establishedsomerupturesthatenabled

freedomforcontentcreationexpandingthelabelof“artist”toalmostanyone.Theyallowed

aninteractivewayofcommunicationbetweenpeople,similartowhatweliveonwith

digitalmedia.Thestrengthoftheirworkalotoftimesliemoreontheconceptualshiftsthan

onactualauralwonders,especiallyifwethinkabout,forexample,JohnCage´s4`33piece.

Thisworkproposesaquestioningaboutwhatmusicis:Ismusicorganizedsound?Chaotic

soniceventscan´tbemusic?Whatdefinesmusic:Rhythm?Melody?Harmony?Thepiece

alsoinvitesthelistenertogainconsciousnessabouttheenvironment,abouteveryday

sounds.Isthereexpressivecontentwithinurbansoundsliketraffic?Coughs?Footsteps?Is

musicmadeonlywithinstruments?4´33alsobreaksadistinctionbetweenartistand

audienceandsetsthembothinasameperceptuallevel.Theconstantprobingabout

apparentlyestablishedconceptsandideasaroundartpromotedarupturethatcanbe

tracedbacktosoundartinthe60sandfollowedintothedevelopmentsofdigitalmediaand

theemphasisonusercreatedcontent.Thequestioningofconceptsestablishedby

26

contemporaryartistsdidn´tremainconstrainedtoacriticismtoinstitutionalizedspacesor

ideas.Itnecessarilydirectedthecenterofactiontothepeople.Iftherearenoinstitutions

thatdefineartorsocialstructuresthenwhodefinesthem?Thepeople.Anyone.Everyone.

Theinclusionofinteractionwithinsoundartsuggestednotonlyadestructionofprevious

ideasbutalsoaconstructionofnewworlds.Theartistscreatedaplatformforuserstostand

onandbecometheirowncreators.Thelinkwithvirtualrealitycanbemadehere,as

immersionandinteractionarestrongelementsforthismedium.Invirtualreality,

immersionisnotonlycreatedbytheincorporationofuserswithintheworkbutalsobythe

interactioninwhichtheyengagewithinthesesurroundingelements.Sincethereisa

connectionbetweensoundartandVR,itisinterestingtoseewhatnewpossibilitiesthis

newmediumofferstoexpandsonicartconcepts.

Asexplainedintheprevioussection,theuseofthebodyisveryimportanttoenhancethe

feelingofpresence,ofstrongimmersionwithinthevirtualworld.Theuseofthebodyis

stronglyrelatedtogestureandeventhoughthisispresentinsoundartprojects,itcanbe

takenintoamuchmoredeeperlevelwithindigitaltechnology.AingerandSchroeder

(2014)writethataninterpreterfeels,listensandwatchesthesoundhemakes.Thismeans

thatsoundisnotrelatedonlytothesoundwavepropagatingthroughtheairandintothe

earbutitisalsorelatedtomoresenses:itinvolvesamorecomplexmultisensorycognitive

system.AingerandSchroeder(2014)alsostatethattheuseofphysicalmodeling,withinthe

digitalrealm,dividesthequalitiesofsoundindifferentparameters,andthroughthis,itis

possibletoassigndifferentcharacteristicsofsoundtodifferenttypesofgestures,aswellas

assignpartialcontrolofthemalsotocomputers.Althoughintraditionalsoundartprojects

itwaspossibletochangesonicparametersdependingonactionsoftheparticipants,these

sonicchangeswerelimitedbecauseweweredealingwithrealsounds,notmodeledones:

soundisnotdividedinveryspecificparametersbyphysicalmodelsordigitizedversionso

thesecan´tbeattacheddirectlytoactionsmadebytheparticipant.Thelevelofdetailgiven

bythedigitalrealmisoneofthemainelementsthatgiveVRanewdimensiontosonic

interactionandimmersion.

Virtualrealityisamediumwheretheideasofsoundartcanbeexpanded,asseen

previouslywithinteraction(agency).Itcanalsoexpandtheideaofspacetransformations

foundwithinsoundarts,withdigitalmodelingofshapes,textures,andorganizationof

27

architecturalstructures,byapplyinganytypeoftransformationtothesevarious

characteristics.Ifweimagine,forexample,theXenakisandVaresecompositionsforthe

PhilipsPavillionin1958,thewholeconstructionofthestructurewasintendedtoaffectthe

waysoundwaspropagatedandperceivedbypeople.Thisincludedaspatializationofsound

basedonloudspeakerplacementthroughoutthebuilding.Invirtualreality,buildingscanbe

madewithouthugeamountsofmoneyortimeandinanyimaginedshape.Theformsand

texturesofbuildingscanbechangedtomodelcompletelynewpropagationmediumsfor

sound.Thewallscouldbemadeofwaterandthefloorsofsandorviceversa.Thespeedof

soundcanbealteredaswellastheabsorptionqualitiesofmaterialsrelatedtofrequency

responses.Thelevelofdetailintowhichcreatorscansubmergeinwithinthedigitalworldis

almostendless.Thereisnolimitationwithinthevirtualworld,comparedtothephysical

limitationsofprojectscreatedinthe60s,exceptcomputingresources,whichareyearby

yearexponentiallyexpanding.AsKastbauer(2014)wrote,audiocontent,ingeneral,willbe

basedmoreonphysicalmodelingandproceduralmodels.Inthedigitalworld,therealityof

theworldwillbereconstructedthroughveryprecisealgorithms,givingcreatorsandusers

thepossibilityofhavingultrarealisticexperiencesbutalsoenablingtheopportunityof

transformingcompletelythisrealityandcreatingworldsthatarenon-existentorthat

wouldbeimpossibletofindwithinreality.Thephysicalmodelsletusdescribethereal

worldandbecauseofthis,itletsustransformit(AingerandSchroeder2014).Ifwetake

intoaccountnotonlytheseinternalspatialreconfigurationsbutalsosoundplacementmade

throughspeakersininitialsoundartworks,sonicobjectswithinvirtualworldscanalsobe

placedinanydesiredposition,evenbreakingphysicallimitationspresentinreallife.Ifwe

wanttoplaceaspeaker(oranysoundsourceingeneral)insideawallcreatedofwater,isit

easytodoitinreallife?Althoughitmightbefeasibletofindasolutiontothis,thedigital

world,basedonanumericcharacterizationofreality,canofferafasterandmoreefficient

wayofcreatingaudiovisualexperiencesthatgobeyondthephysicalreality,orthattake

reality´scharacteristicsandtransformstheminradicalways.Asitwasseenpreviously,

soundartalsowasinvolvedinreconfigurationsofexteriorandpublicplacessoitistimeto

seehownewdigitaltechnologycandevelopthisarea.

Besidesthinkingaboutinternalspacesbeingchanged,itisalsointerestingtoexplore

alterationsofpublicspaces,withvirtual,augmentedandmixedrealitytools.Asstated

before,inthecontinuumwefinddifferentlevelsofvirtualitydependingonwhatplaceof

28

thespectrumonesituates(seepreviousFigure6).TechnologieslikeOculusRiftandHTC

VivearefullvirtualworldsbuttechnologieslikeMicrosoft´sHololensoffermixedreality

experiences.Theideahereistoblendrealworldmaterialswithdigitalsyntheticones

(Microsoft2016).Insoundart,wehadtranslocationofenvironmentsthatcreatedillusions

ofnewspacesforthelistenersandalsoexteriorprojectslikeRecycledSoundscapes(2004),

whichalteredthepublicspaceandchangedbehaviorsandperceptionsofcitizens.InMR

andAR,itispossibletoaggregatenewsoundstoanestablishedrealenvironmentwiththe

useof3Daudio.Theideaofanalyzingourperceptionandthewaysoundpropagatesin

space,givesnewtoolsthatmakeeasiertheblendingoftwoormoresonicsoundscapesand

makethemseemasone.TomakeaconnectionbetweensoundartpublicspacesandVR,AR,

andMR,itisusefultothinkaboutthepreviousprojectdescribecalledCityLinksand

modernaugmentedrealityprojectslikeWARA(Wearableaugmentedrealityaudio).With

thisproject,specialheadphonesthatincorporatemicrophoneswereusedtoblendthereal

environmentandvirtualsounds.Throughsoundspatialization,anillusioncouldbecreated

toplacesyntheticsoundsnexttorealworldonesandmakethemseemasone.(Harma,

Jakka,Tikander,Karjalainen,Lokki,Nironen,Vesa,2003).3DaudioisusedinVR,ARand

MR,and,dependingonthetechnologyused,itenablesatotallysyntheticenvironment(VR)

oramixedonewithrealityanddigitalelementscombined.Thiscreatesahugerangeof

possibilitiesthatwerenotpossibleinthe60sforsonicartistssinceitletsanindividualto

immerseitselfinanyenvironmentcreateddigitallyanditcanalsoincorporatereal

elements.Itisinterestingalsotonote,thattheseapparatusesaremobile,andthismeans

thatapersoncanrecreateitsownenvironmentnotonlyinaspecificcontextorplacebut

anywhereandanytime.Themobilityenabledbynewtechnologyisahugeadvancement

whenitcomestogivingthepeoplethepossibilitiesofself-creativeexplorationthrough

immersionandinteraction.Therupturewithstaticartmuseums,galleries,andexhibition

spacesthatwasoncepromotedbysoundart,isnowexpandedbydigitaltechnologies.

Onelastimportantaspecttodiscussisthepossibleinteractiveusesofsoundsused

traditionallyinfilm:Dialogue,musicandsoundeffects.Therearenewcreativepossibilities

withthemergingoffieldslikefilmandgames.MateasandStern(2005)developedan

interactivedramacalledFaçade.Inhere,thestoryisdrivenbythemoment-by-moment

interactionoftheplayerandthenarrativeevolveswithdialogueenhancedwithAI(artificial

intelligence).Theuserinputsdialoguethroughakeyboardand,throughAI,thestory

29

evolvesdependingonwhattheplayersaid.Ifvoicerecognitionisaddedtothis

environment,thenaricherinteractioncanbeachieved.Thereconstructionofmeaningand

emotionalintentioninteractingwithdialoguesandmusicisanareathatisstillnotstrongly

developed.Thisnewmediumopenspossibilitiesofinteractionwithfilmicelements,which

couldincludesounds,voices,andmusic.Imagineaninteractivefilmwheretheparticipant

(inthiscasethewordaudiencedefinitelylosesvalidity)isabletochangenotonlythe

dialoguebutalsothemusicthatunderscoresamoment.Itcangivethepersonatotal

creativecontrolaboutwhatcouldhappenwithinastory.Itisusefultothinkabout

traditionalusesofsoundtoseehowthey,withininteractivecontexts,canchangethe

contentandopencreativepossibilitiesfortheplayers.

2.5 Conclusion

Toconcludethischapterisusefultosummarizethetwomainelementsthataretranslated

fromsoundarttoVR.Thesetwoareimmersion(space)andinteraction(agency).

Whentalkingaboutimmersion,soundartsexploredtheuseofspatializationofsound

throughtheuseofmultipleloudspeakerarrangementsandreconfigurationsofsonic

emitterscreatingamergingoftheparticipantwiththepieceofart.Soundartalso

incorporatedinteraction,givingparticipantsasenseofagencywhenbeinginvolvedwithart

pieces.InVR,spatialauralenvironmentsthrough3Dsoundexpanded,evenmore,theidea

ofimmersionaswellastheinclusionofinteractionwithin,forexample,videogames.The

possibilitiesofalteringthesonicelementsofasoundscapecreateahigherlevelofpresence

andthereforeimmersionwithinavirtualworld.ThefollowingcasesofVRwillevidence

howimmersionandagencyarepresentinthemandhowtheyrelatetosoundart.

Chapter3

3.1Introduction

SoundartprojectshavecharacteristicsthatcanbeoverlappedandexpandedinVirtual

Reality.Thefollowingfourcasescontainspatial,andinteractivecharacteristicsthathave

beenmadeinVRprojectsandwiththemwe´llseehowsoundartcharacteristicsarealso

present,showingtheconnectionbetweenthetwofields.

30

3.2VRProjects

3.2.1GhostbustersTheVoid

TheVoidGhostbustersisa“hyper-reality”projectwhereplayersareimmersedinaphysical

space,overlaidwithdigitalcontentthroughvirtualreality.Theexperienceisanimmersion

expandedbyawirelesssystemthatallowsplayerstomovefreelythroughadigitalworld.

Throughoutthegame,theplayersareaffectedbysensorialstimulithrough3Dsound,

stereoscopicvisuals,theuseofahapticvest,heat,air,andmistthatfollowtheactionsofthe

virtualworldandcreatesafullyembodiedexperiencefortheusers(Wired2017).This

projectisveryinterestingbecause,similartoCamileUtterback´sVicissitudes,thebodyand

itsmovementbecomesessentialforthecreationofthepiece.Theimmersioniscreatedby

theinclusionofmoresensesbesidesvisionandaudition,andthemovementofthewhole

bodygivestheuserahighsenseofagencyandimmersion.Theplayerisnotonlyableto

movehisheadaroundtheworldbutalsotointeractdirectlywithphysicalobjectsthathave

arepresentationinthevirtualworld.Userscan,forexample,grabafirelampthatisareal

worldartifactandinthiswaysenseitthroughtouch(seefigure13and14).

Fig.13TheVoidphysicalobject(Source:Wired.YouTube.2017)

Fig.14TheVoidvirtualobject.(Source:Wired.YouTube.2017)

31

Therealityofthevirtualworldisstronglyenhancedbylettingplayersfeeltheobjectsof

thevirtualworld,asstatedbyWired(2017).Similartosoundartprojects,theplayerisnow

insidethepieceofart,breakingthebarrierseparatingtheworkandtheaudience,anditis

interactivesincetheactionsoftheplayerareaffectingthecontentcreatedinthegame.

CurtisHickman,co-founderofTheVoid,statesthatrealityisperceivednotonlythroughour

visionandthatinthisproject,thecreationofanewworldisachievedbyincorporating

moresensestotheexperience,allowingtocreateacompletelynewrealitybutwhere“we

notonlymakearegularreality,wemakeanimpossiblerealityforyoutoplay”(Wired

2017).Thisisveryimportantifwegobacktotranslocationideaspresentinsoundart

projectslikeCityLinks.Inthatproject,newsonicenvironmentswerecreatedandmoved

fromoneplacetoanother,andsimilartoTheVoid,anewsenseofrealitywasestablished

bycreatingnewsoundscapes.Thisexperienceoffersalsoanotherveryinterestingaspect

thatrelatestosomeofthereconceptualizationpromotedbysoundart.AsSmall(1998)

stated,thedesignoftraditionalconcertauditoriumscreatedaspacewheretheinteraction

betweenmembersoftheaudienceswasdiscouraged.Contemporaryartbrokethis

tradition,forexample,withprojectslikeDriveinMusicorLines(2016),aprojectcreatedby

AndersLindinwhichagroupofpeopleisabletointeractwithsonicelementsatthesame

time.Ghostbustersalsoallowsaninteractivesocialexperiencesincethreepeoplecanplay

thegameatthesametime.Theyareallimmersedandactingtogetherwithinthevirtual

world.

Ingeneral,theenvironmentsinthesenewvirtualworldsarebecomingmoreinteractive

andthroughthis,theplayersareallowedtoimmersethemselvesevenmoreintheworld

thanwithprevioustechnologies.Thefollowingprojectshowssomemoreofthese

interactivesonicaspectsinVR.

3.2.2PlayStationVRWorlds

Thisprojectinvolvesfivedifferentscenariosthattheplayercanplay.

1. TheLondonHeist:thisworldsetyouinsideamafiainLondonthatincludesdriving

andshootingmechanics.

32

2. OceanDescent:thisgameisbeneaththeseaanditletsyouexploretheunderwater

sceneryfromwithinacage.Itisamoreexperientialenvironmentbutstillhas

interestinginteractivesections.

3. ScavengersOdyssey:thisVRworldtakesyouonanextraterrestrialspaceshipintoa

nebulahuntingforanoldartifact.Itisashootinggamefightingaggressivealiens.

4. VRLuge:Thisgameisbasedonillegalroadracing.Itisexperiencedasaperson

layingonaboardanddivingdownhillwhileavoidingtrucks,carsandmanymore

typesofdangerousobstacles.

5. DangerBall:ThisisagamesimilartoPing-Pongbutwithina3Denvironmentwhere

youhitaballagainstdefinedtargets.

Whenitcomestointeractivityandimmersionitisinterestingtoseesomespecific

characteristicswithinthesegames.ThefirstremarkablecharacteristicispresentinOcean

descent.Eventhoughitisagamethatisbasedmoreonobservationthanonheavyaction

madebytheplayer,interactionandimmersionwerecreatedbyshiftingmixinglevelsof

sonicelementsbasedonthepointofviewoftheplayerandaddingsubtlesoundstothe

movementsoftheavatar.AsSimonGumbletonsaysinthePlayStationEU(2016)video,one

oftheessentialideasofVRisto“groundtheplayerinthatspace”.Thismeanstocreatea

feelingofpresencewithinavirtualworld.Itisveryinterestingtoseethatbesidesthe

spatializationofsoundwith3Dsound,thesounddesignersdecidedto,first,addsoundthat

wasgeneratedwitheveryshiftofperspectiveoftheplayer,creatingthesoundofthe

movementoftheheadinthewater,and,second,theycreatedadynamicauralenvironment

where,afteracoupleofsecondsoftheplayerfocusing,forexample,inadistantturtle

swimming,thelevelofthesoundsofthisanimalstarttoraisewhileotherenvironmental

soundsgodown(seeFigure15).

33

Fig.15OceanDescent(Source:PlayStation[online].2016)

Thismeansthatinteractionisbasedontheplayer´sheadmovementandonitsperspective.

Theauralenvironmentisnotrigidnotonlybecauseitisspatializedandtheheadtracking

changestheperceptionofthesoundsdependingonthepointofviewoftheplayer,butalso

becausethefocusoftheplayeristakenintoaccountandchangessoundlevelshighlighting

orhidingsonicelements:interactionnotonlygivesasenseofagencytotheplayerbutit

alsoenhancesimmersionbymakingtheenvironmentreacttoitsdecisions.

AnotherinterestingtraitofVRworldsistheuseofthebuilt-inmicrophoneoftheheadset.

AsJonasAndresJensenstatesinPlayStationEU(2016),theplayersnowareabletointeract

alotmorewiththeenvironment.Intraditionalgames,healsostates,theinteractionwith

objectswouldbemadethrough“fixedanimations”whilenowtheplayermanagesinhis

owntimetheinteractionwiththeobjects.Asanexample,hedescribesasectionwherethe

playerlightsacigarandthroughthemicrophone,theuserisabletoactivateinhaleand

exhaleactionsoftheavatar:theactionismade“atyourownspeed”.Anotherexampleofthe

useofthemicrophoneaffectingthevirtualworldisinthemenuwhereyoucanscreamand

thiswillaffectfloatingrocksthataremovingaroundaplanet(seeFigure16).Theuseof

anotherinputsensorbesidesthehandsandheadhelpstheplayertoincreaseits

possibilitiesofinteractingwiththeworldandcreatingthefeelingthathe/sheisactually

presentinthatenvironment.

34

Fig.16PlayStationVRWorldsMenu(Source:PlayStationEU.YouTube.2016)

3.2.3Playthings:MusicalVRPlayground

ThisprojectwascreatedbyAlwaysandForever.Itconsistsofavirtualworldwhere

hamburgers,hotdogs,gummybearsandmanyotherfoodtypesareusedasmusical

instruments.Throughtheheadsetsandtheuseofcontrollers,theplayersareabletocreate

musicbyinteractingwiththeseobjects(seeFigure17).

Fig.17PlaythingsGameplay(Source:AlwaysandForever.YouTube.2016)

Itisinterestingtoseeinthisprojecttwoaspectsinrelationtosoundart.Ononehandthe

ideaofenablinganypersontobepartofthisworldandaffectit“whetherornotyouarea

musicianoragamer”(AlwaysandForever2016).Thiscoincideswiththeideasofrupture

betweenartistsandaudiencespresentinsoundart,bygivingthepowertothepeoplefor

creatingtheirownexperiences.Insoundart/experimentalmusicprojects,likePlaythe

Building,non-musicianswereabletomakemusicwithabuilding´sstructureasan

instrument.IfwethinkaboutPlaythingsVRwecanseethesesameelements.First,anon-

musicalobjectistransformedintoamusicalinstrument(virtualfoodandbuilding)and

second,thepossibilityofartisticcreationisgiventoanyone,nomatteriftheyareartistsor

35

not.VRandSoundArtsharecommonconceptsandtheyareevidentinthiscasethrough

interaction:givingpeopleasenseofagency.

3.2.4GnomesandGoblins

Thispieceisstillaprojectunderdevelopmentbutitisinterestingbecauseitisamixtureof

filmandgame:ofinteractivityandlinearmedia.Thegeneralstoryisaroundtwoscenarios,

onewithgnomesandonewithgoblins,andtherelationshipscreatedbytheplayerwiththe

beingsofthesetwolocations(seeFigure18).

Fig.18GnomesandGoblins(Source:TransportVR.2016)

AsreportedbyMurphy(2016),thecreatorJonFavreaustatedthattheideaistomakea

mixoffilmandgamesexploringtheideaofimmersion.Whatisinterestingtoexplorehere,

forFavreau,istofindabalancebetweengivingcontroltotheplayertoaffectthestoryand

environment,andatthesametimecreateadefinedplotthatguidestheplayer´sfocus:the

roleofthedirectorstillexistsbutwithnewrules.Oneofthemostimportantaspectsofthis

projectisartificialintelligence(AI)sincetherelationshipcreatedwiththegnomesiscentral

tothestory.Infact,thewayapersondecidestointeractwiththegnomeaffectshowthe

storyisdeveloped.Inthisproject,interactionispresentnotonlybyaffectingtheworld´s

environmentalobjects(grabbingfruits,openingdoors,ringingbells)butthroughthe

interactionwithcharacters.Whendescribingthetypeofinteractionwiththegoblinshown

inthepreviewofthegame,Favreaustates,“it´snotachecklistofwhatyouhaveto

accomplish,butthewayyouapproachhim,howcloseyouare”(Murphy2016).Ifwethink

aboutsoundart,theideaofco-creationpromotedbypreviousartistsiscompletely

36

applicabletothisproject.Thedirectorissettingaworldwhereaplayercanmoveandact

affectingtheoutcomeofthestorybutstillpushinghim/shetowardsaspecificgoal.Here,

theprojectiscreatedbothbyadirectorthatplotsanopenstoryandaplayerthatmoves

andredefinesthisstorydependingontheactionsandbehaviorsithas.Sinceitisstilla

projectunderdevelopmentitwillbeinterestingtoseeifdialogueenhancedwithAIoreven

musicinteractionisincludedtoimmersetheplayersevenmoreinsidethestory.

3.3Conclusion

TheVRprojectsexposedinthissectionshowsomeofthelatestcreationsmadeinVR.They

evidencethedegreeofinteractivityandimmersionthathasevolvedsincethebeginningsof

thistechnology.Interactivesoundscapesandmultisensorysystemsarepushingthe

boundariestocreatemorerealisticalternativeexperiencesthatgivetheuseragreater

senseofagencyandthereforeimmersion.Itisthroughthesecaseswhereweseeanew

perspectiveonimmersion.Inpreviousexamplesofsoundarts,itwasstatedthatonething

wasimmersion,thefeelingofbeingpresentinagivenspace,andanotherwastheabilityto

changethecontentwithinthisworld,whichreferredtointeraction.Herethereisa

connectionbetweenthetwotermsnotonlybycoexistencewithinvirtualrealitybutalsoby

areinforcementofonebyoftheother.InteractioninVR,throughsonicelements,further

enhancesthefeelingofbeingpartofanewworld.Spatialreconfigurationsandinteractivity,

whichwerestronglydevelopedbysoundarts,aretakenintonewlevelsofelaborationin

VR.

ItisinterestingtoseehownewcreativedevelopmentswillgrowwithinVR.Themedium

couldtakeelementsfromfilmandgamestoinnovate.Infilm,forexample,theideaofpoint

ofviewisveryinterestingsinceitisusedtoshowacertainperspectivewithinastory.In

thismedium,forinstance,theequalizationandloweringofthelevelofenvironmental

soundsisatechniqueoftenusedtosuggestanimmersionwithinthemindofacharacter.

Thiscouldbeincorporatedwithinteractivedramastomakeshiftsincharactersand

positionswithinstories.SinceVRsuggestsalmostimmediatelyafirstpersonperspective,

theuseofvariationsofsoundcouldchangethispointofviewtoshiftanglesandcreate

possiblethirdpersonperspectivesorchangesintodifferentcharacters.Expandingonthe

ideaofpointofview,ifwetakeasanexamplethefilmBeingJohnMalkovich(1999),wecan

37

thinkofanimmersionwithinacharacterwherewehearandseeeverythinghedoes,while

stillhavingourownconsciousness.SoundinVR,asinfilm,couldofferthechancestoshift

betweeninternalselfdirecteddialoguesandexternalcommunicationswiththeworld.VR

worldscouldbeenhancedbyincorporatingself-consciousnessandexteriorcommunication

withcleverusesofspeechwhere,throughsoundchangeswithequalizationand

spatialization,theplayercangainanevendeepersenseofpresence.Theavatarwithinthe

virtualworldisnotonlyaphysicalbeingthatcanbecontrolledbutalsoaconsciousone:it

canhearhisinternalmindvoice.

VRcouldalsoincorporateideasfromvideogamestoinnovate.Theideaofcompetitionand

rewardbasedonpointsisacommonelementinmanygames.Theideaofcompetitionand

rewardcouldbeusedwithinVRexperienceslikeinteractivefilmswhere,besideshavingthe

freedomtoactwithinavirtualworld,andfollowingapredefinedstory–drivenexperience,

theuserscanengageininternalcompetitionsdependingonspecificgoalscreatedwithin

thestory.

ThecreativepossibilitiesofVRwillcontinuetogrowasthetechnologybecomesmore

popularandavailableforeveryonetouse.Onthehandsofthecreatorsofcontentsliesthe

responsibilityofboostingthemedium,byofferingcompellingandnewinteractive

experiences.ItisusefultoseetechniquesusedinothermediumstoexplorethemwithinVR,

assomeofthepreviousexamplesshowed.

Chapter4

GeneralConclusion

ThisworkhasmadeevidentatranslationofconceptsbetweensoundartandVR,focusing

especiallyonimmersionandagency.Immersionwaspromotedinsoundartthrough

alterationsofspace,usingsonicreconfigurationswithloudspeakersthatsurroundan

audienceandtranslocationsofauralenvironments,amongmanyothertechniques.

Envelopingacousticcreationswerealsodevelopedwithinfilmwiththeuseofsurround

systemslike5.1,7.1andDolbyAtmosandfinally,morerecentdevelopmentsinvolving

spatializedauralconstructions,whichtakeimmersiononestepfurther,arepossiblewith

38

theuseof3DaudiousedinVR.Theideaofmergingtheaudienceandthepieceofart,which

waspromotedfromthe60sinsidecontemporaryart,isstronglyexpressedandheightened

inVRwith3Daudiobutalsowithinteraction.Soundartalsoencouragedtheparticipation

ofpeoplewithinpiecesofart,breakingdichotomiesbetweenartistsandaudiencesand

establishinginteractive,two-waycommunicativesystems.Similarly,interactivemediums

likevideogamesandinteractivedramas/moviesinVRinvolvetheparticipationofplayers

thataffectthecontentofthemediumandpromoteacommunicativeloopfeedbackbetween

thedigitalworldandtheinputoftheparticipants;atwo-waycommunicativestructure.

Throughinteraction,asenseofagencyisgiventoparticipantsbothinsonicartprojectsas

wellasinVRones.Throughthispaper,itwascreatedatranslationofconceptsfromsound

arttoVR,anditwasalsoexploredhowaudioininteractivemediaisused.Besidesthis,it

gaveageneraloverviewoftheuseofsoundintraditionalmediatoseehowitcouldbeused

withinthisnewdigitalmedium.

StephenO’Callaghan,headofSound,Tools,andTechnologyinPlayStationVRWorlds,said

“themainchallengeforusinworkingwithVRasamediumwasthatwewereonuncharted

territory;therewasnohow-tomanualforaudio”(PlayStationEU2016).ThefieldofVR,AR

andMRissonewthattherearenopredefinedguidesofhowaudioshouldbeused,and

eventop-levelprofessionalsareinexploratorystages.Alotofresearchhasbeendoneto

achievetechnicaldevelopmentsbutnowistimetousethetechnologyincreativeways.

Hopefully,thisworkcontributestopromotenewwaysofthinkingaboutaudiotaking

motivationsfrompreviousbrilliantsoundartistswhotranscendedtraditionalideasand

empoweredtransformationsincreativeworks.

39

References

Ainger,M.,SchroederB.2014.GestureintheDesignofInteractiveSoundModels.TheOxford

HandbookofInteractiveAudio.OxfordUniversityPress.

Altman,M.,Krauss,K.,Susal,J.,Tsingos,N.2016.ImmersiveAudioinVR.AESE-library

[online].AESConferenceonAudioforVirtualandAugmentedReality.LosAngeles

September30-October12016.Availablefrom:http://www.aes.org/e-lib.

AlwaysandForeverComputerEntertainment.2016.PlaythingsVR.[online]Availablefrom:

http://playthingsvr.com/[Accessed7May2017]

Auro-3D.2017.Auro-3D.[online]Availablefrom:http://www.auro-3d.com/professional/a-

new-standard/[Accessed29April2017]

BeingJohnMalkovich.1999.[film].DirectedbySpikeJonze.USA:PropagandaFilms,Single

CellPictures,GramercyPictures.

Bordwell,D.,Thompson,K.,2012,FilmArt_AnIntroduction,10thedn,McGraw-Hill,New

York.

Byrne,D.2005.PlaytheBuilding.DavidByrne[online].Availablefrom:

http://journal.davidbyrne.com/explore/playing-the-building[Accessed8May2017]

Chion,M.1994.Audio-Vision:soundonscreen.ColumbiaUniversityPress,NewYork.

CNET.2016.TheHistoryofVR[video,online].YouTube.Availablefrom:

https://www.youtube.com/watch?v=hDWfsIbBtwE[Accessed30April2017]

40

C.S.W.2013.GogglesforGamers.TheEconomist[online].3July2013.Availablefrom:

http://www.economist.com/blogs/schumpeter/2013/07/oculus-rift[Accessed30April

2017]

Digi-Capital.2017.Aftermixedyear,mobileARtodrive108billionAR/VRmarketby2021.

[online].

Availablefrom:http://www.digi-capital.com/news/2017/01/after-mixed-year-mobile-ar-

to-drive-108-billion-vrar-market-by-2021/#.WPPwelOGORs

[Accessed22April2017]

Dolby.2017.DolbyAtmos.Availablefrom:https://www.dolby.com/us/en/brands/dolby-

atmos.html[Accessed27April2017]

ForensicVR.VR.Availablefrom:http://forensic-vr.com/VR[Accessed30April2017]

Franinovic,K.,Serafin,S.2013.SonicInteractionDesign.TheMITPress:Cambridge.

Franinovic,K.,Salter,C.2013.TheExperienceofSonicInteraction.SonicInteractionDesign.

TheMITPress:Cambridge.

Gervautz,M.,Mazuryk,T.1996.VirtualrealityHistory,Applications,TechnologyandFuture.

ViennaUniversityofTechnology.Availablefrom:

https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/[Accessed

26April2017]

Gigante,M.1993.VirtualReality:Definitions,HistoryandApplications.In:Earnshaw,R.,

Gigante,M.,Jones,H.eds.VirtualRealitySystems.AcademicPress:London,3-14.

Harma,A.,Jakka,J.,Tikander,M.,Karjalainen,M.,Lokki,T.,Nironen,H.,Vesa,S.2003

Techniquesandapplicationsofwearableaugmentedrealityaudio.AES[online].AES114th

Convention.Amsterdam22-25March2003.

41

Hermes,W.2000.TheStoryof4’33.NPR[online].Availablefrom:

http://www.npr.org/2000/05/08/1073885/4-33

[Accessed26April2017]

Herzer,J.2014.AcousticScenographyandInteractiveAudio:SounddesignforBuilt

Environments.TheOxfordHandbookofInteractiveAudio.OxfordUniversityPress.

Hook,B.2014.OculusConnect:IntroductiontoAudioinVR.YouTube[online].Available

from:https://www.youtube.com/watch?v=kBBuuvEP5Z4&t=1749s

[Accessed1May2017]

Kastbauer,D.2014.OurInteractiveAudioFuture.TheOxfordHandbookofInteractiveAudio.

OxfordUniversityPress.

Lamkin,P.2017.BestVRHeadsets2017:HTCVive,Oculus,PlayStationVRcompared.

Wareable[online].14March2017.Availablefrom:https://www.wareable.com/vr/best-vr-

headsets-2017

[Accessed30April2017]

Leitner,B.1995.SoundFieldIV.BernardLeitner.Availablefrom:

http://www.bernhardleitner.at/works[Accessed8May2017]

Licht,A.2009.SoundArt:Origins,developmentandambiguities.OrganisedSound,Volume

14(1).

Lind,A.2016.LINES-aninteractivesoundexhibition.YouTube[online].Availablefrom:

https://www.youtube.com/watch?v=hP36xoPXDnM&feature=youtu.be[Accessed10May

2017]

Ludlow38.2010.MaryanneAmacher:City-Links.GoetheInstitut,NewYork.Availablefrom:

http://ludlow38.org/archive/index.php?/interviews/maryanne-amacher-city-links/

42

Mateas,M.,Stern,A.2005.StructuringcontentintheFaçadeInteractiveDrama

Architecture.ProceedingsoftheFirstArtificialIntelligenceandInteractiveDigital

EntertainmentConference.

Microsoft.2016.MicrosoftHololens.Availablefrom:

https://www.microsoft.com/microsoft-hololens/en-us/why-hololens.(Accessed5of

December2016)

Milgram,P.Takemura,H.Utsumi,A.andKishino,F.1995.Augmentedreality:aclassof

displaysonthereality-virtualitycontinuum.

Murphy,M.2016.JonFavreauon’GnomesandGoblins’andVirtualReality.NewYorkTimes

[online].30August2016.Availablefrom:

https://www.nytimes.com/2016/08/31/movies/jon-favreau-on-gnomes-goblins-and-

virtual-reality.html[Accessed8May2017]

Neuhaus,M.1993.NotesonPlaceandMoment.Availablefrom:http://www.max-

neuhaus.info/soundworks/vectors/place/notes/[Accessed26April2017]

Oculus.2017.LocalizationandtheHumanAuditorySystem.Oculus[online].Availablefrom:

https://developer3.oculus.com/documentation/audiosdk/latest/concepts/audio-intro-

localization/[Accessed1May2017]

Oldenburg,A.2013.SonicMechanics:audioasgameplay.TheInternationalJournalof

ComputerGameResearch.Volume13(1).Availablefrom:

http://gamestudies.org/1301/articles/oldenburg_sonic_mechanics

[Accessed24April2017]

PlayStation.2017.VRWorlds.PlayStation[online].Availablefrom:

https://www.playstation.com/en-ie/games/vr-worlds-ps4/[Accessed7May2017]

43

PlayStationEU.2016.TheSoundandMusicofPlayStationVRWorlds.YouTube[online].

Availablefrom:https://www.youtube.com/watch?v=y5FLTyRMpOM&feature=youtu.be

[Accessed7May2017]

Rogers,H.2014.SpatialReconfigurationinInteractiveVideoArt.TheOxfordHandbookof

InteractiveAudio.OxfordUniversityPress.

Schoenherr,S.2005.RecordingTechnologyHistory.AES.[online].Availablefrom:

http://www.aes.org/aeshc/docs/recording.technology.history/notes.html[Accessed27

April2017]

Small,C.1998.Musicking:TheMeaningsofPerformingandListening.Middleton,CT:

WesleyanPress.

Solomon,B.2014.FacebookbuyOculus,VirtualRealitygamingstartup,for$2billion.

Forbes.[online].25March2014.Availablefrom:

https://www.forbes.com/sites/briansolomon/2014/03/25/facebook-buys-oculus-virtual-

reality-gaming-startup-for-2-billion/#73a430d32498

[Accessed23April2017]

Sonnenschein,D.,2001.SoundDesign.Theexpressivepowerofmusic,voice,andsoundeffects

incinema.MichaelWieseProductions,LosAngeles.

Stevens,T.2011.NintendoVirtualBoyreview.Engadget.[online].Availablefrom:

https://www.engadget.com/2011/03/21/nintendo-virtual-boy-review/

[Accessed22April2017]

Tanaka,A.Interaction,Agency,Experience,andtheFutureofMusic.InBrown,B.,O´Hara,K.

ConsumingMusicTogether:SocialandCollaborativeAspectsofMusicConsumption

Technologies.ComputerSupportedCooperativeWork(CSCW)Vol.35.Springer,Dordrecht,

2006.

44

Tittel,C.2009.SoundArtasSonification,andtheArtisticTreatmentofFeaturesinour

Surroundings.OrganisedSound,Volume14(1).

TheEconomist.2015.PlanetofthePhones.TheEconomist.[online].26February2015.

Availablefrom:http://www.economist.com/news/leaders/21645180-smartphone-

ubiquitous-addictive-and-transformative-planet-phones

[Accessed22April2017]

TransportVR.2016.Gnomes&Goblins.Availablefrom:

https://www.transportvr.com/gnomes-goblins[Accessed8May2017]

Utterback,C.2004.ElectronicBookReview.[online].Availablefrom:

http://www.electronicbookreview.com/thread/firstperson/multitiered[Accessed27April

2017]

Wired.2017.StepIntotheVoid,WhereVRMergeswiththeRealWorld.YouTube.Available

from:https://www.youtube.com/watch?v=WUs5qi_RFnM

[Accessed7May2017]

Zero-Th.2004.RecycledSoundscapes.Zero-th[online].Availablefrom:http://www.zero-

th.org/RecycledSound.html[Accessed8May2017]