Andreas FlorosDept. of Audiovisual Arts,
Ionian University
Nicolas -‐ Alexander TatlasDept. of Electronic Engineering,
Technological Educa?onal Ins?tute of Piraeus
Stylianos Po?rakisDept. of Electronic Engineering,
Technological Educa?onal Ins?tute of Piraeus
Sonic Perceptual Crossings: A 4c-‐tac-‐toe Audio Game
AudioMostly’11, September 7–9, 2011, Coimbra, Portugal
Introduc?on
• Video -‐ game concept:
-‐ The visual component is cri?cal for the game-‐play
-‐ Audio informa?on is used (mainly) for the ambient environment synthesis
• Audio -‐ games
-‐ Increasing research interest
-‐ Sound represents a prominent component for realizing the required human-‐machine interac?on interfaces
-‐ Suitable for specific user target groups, such as visually impaired people
Introduc?on (cont’d)
• A defini?on aVempt:
-‐ Audio-‐games are game applica?ons that employ appropriately synthesized auditory displays for developing the game-‐play scenario and establishing the user-‐computer interac?on
• Sonic design is a key aspect for audio-‐games development
-‐ Design perceptually efficient auditory interfaces
• Sonifica?on
-‐ Sonic design techniques for data (and game-‐world for the purposes of the current work) representa?on
Introduc?on (cont’d)
• The large variety of sonifica?on techniques has allowed the development of mul?ple audio game genres
• We hereby discriminate the following two categories:
-‐ a) audio games evolved based on exis?ng (video) game scenarios
‣ scenario adapta?on may be required, taking into account the narra?ve dis?nc?veness of an auditory environment
-‐ b) audio game scenarios developed from the scratch, targeted to be realized using auditory displays only
Mo?va?on
• This work focuses on the design / realiza?on of the ?c-‐tac-‐toe game
-‐ We use a novel auditory and gestural interface combina?on
-‐ Suitable for execu?on in mobile devices and pla[orms
• We aVempt to focus on sonic design issues:
-‐ inves?gate the performance of an advanced earcons scheme
-‐ op?mize playability performance
• Our approach incorporates a gestural / movement-‐tracking user interface that handles all the user movements and allows a completely eye-‐free game implementa?on
Grid-‐based games pariculari?es
• Wide acceptance: many legacy ?tles exist
-‐ The players are already aware of the applied game rules
-‐ Eliminate any requirements for describing the game-‐scenario details through audio-‐means only
• The game-‐play can be algorithmically described
-‐ allow the employment of a sound design mechanism that takes into account the determinis?c game rules
Applied sonifica?on approach
• Employed sonifica?on strategy: Earcons
-‐ sounds synthesized by combining “fundamental” building sonic mo?ves created using variable sound parameter values
‣ i.e. rhythm, pitch, ?mbre e.tc.
• Earcons pros and cons
-‐ (+) Their synthesis approach allows the representa?on of concurrent (parallel) earcons, provided that specific design rules will be applied
‣ Concurrency can be more efficient provided that spa?al characteris?cs in three dimensions (3D) are incorporated
-‐ (-‐) Since they do not relate to their referent informa?on in terms of the targeted context, user training is required to render them recognizable
Applied sonifica?on approach: Design layer
• Earcons for “X” and “O”
-‐ xij and oij
• Earcons for “X” and “O” placement
-‐ xsij and osij
• empty cell earcon
-‐ emptyij
• i and j are the grid coordinate indices
• boundary0X
-‐ the rolling-‐sto wall auditory icon (will be explained later)
x00 xs00
o00 os00
empty00
x01 xs01
o01 os01
empty01
x02 xs02
o02 os02
empty02
x10 xs10
o10 os10
empty10
x11 xs11
o11 os11
empty11
x12 xs12
o12 os12
empty12
x20 xs20
o20 os20
empty20
x21 xs21
o21 os21
empty21
x22 xs22
o22 os22
empty22
boundary01
boundary03
boundary02
boundary04
Applied sonifica?on approach: Design layer details
• xij and oij
-‐ constructed as simple note mo?ves with variable pitch and rhythm parameters, using the following two rules:
‣ For consecu?vely increasing j-‐index values, the fundamental pitch frequency is doubled
‣ The rhythm of the overall note structure is propor?onal to the i-‐index values
• xsij and osij
-‐ constructed by mixing the original xij and oij earcons with a very short impulsive click sound
• emptyij
-‐ constructed as a smoothed, low intensity and very short dura?on “tapping” sound (the same for all cell-‐posi?ons)
Applied sonifica?on approach: spa?al layer
• All earcons were spa?ally processed using binaural technology
(i,j)=(0,0) (i,j)=(0,1) (i,j)=(0,2)
(i,j)=(1,0)
(i,j)=(2,0) (i,j)=(2,1) (i,j)=(2,2)
(i,j)=(1,1) (i,j)=(1,2)
φ=0ο φ=45ο
φ=90ο
φ=135οφ=180οφ=225ο
φ=270ο
φ=315ο
Applied human control interface
• Basic requirement: eye-‐free control
• The auditory rolling pellet:
-‐ assume the game grid as a two-‐axis revolving flat surface, with a pellet rolling on the top of it
-‐ a rota?on of the grid surface would force the pellet to roll towards the direc?on of the specific rota?on
-‐ the user can control the pellet instantaneous posi?on by simply arranging the two rota?on angles
‣ the rolling velocity of the pellet is considered constant and independent from the rota?on angle in both control axes
Applied human control interface (cont’d)
• The auditory rolling pellet is also aVached to an earcon depending on
-‐ the spa?al posi?on of the cell grid it stands on
-‐ the type of the cell: filled or non-‐filled
• It is reproduced only once (by the moment the pellet enters the specific cell)
-‐ it is replaced by another earcon, only when the pellet enters a different cell
Applied human control interface (cont’d)
• Pellet mo?on tracking
-‐ based on the data provided by a 2-‐axis gyroscope
• “X” or “O” grid-‐cell fill
-‐ the user rapidly shakes the control equipment (an accelerometer is required)
• Grid boundaries determina?on:
-‐ When the auditory rolling pellet reaches the grid boundaries, it “crashes” on a virtual wall causing it to stop rolling
-‐ A corresponding spa?al roll-‐stop auditory icon is then reproduced (boundary0X)
Game architecture (sogware/hardware)
Logic Module
Player 1
Grid Controller
Player 2
Rolling pellet interface
2-axisgyroscope
accelero-meter
Earcons database
Auditory Controller
Binaural Audio Playback
Arduino Physical Computing Interface
Processing software platform
Results -‐ Test methodology overview
• Audio-‐game design and game-‐play assesment
• A two-‐level subjec?ve test was employed
-‐ Level #1:
‣ consider the sonic design efficiency, taking into account the par?cular requirements imposed by the game scenario and rules
-‐ Level #2:
‣ play the game in real-‐world condi?ons
Results -‐ Level #1 tests
• 20 non-‐audio expert adult subjects par?cipated
• All subjects were first given a demonstra?on of the system together with:
-‐ a detailed explana?on of the correla?on of the different earcons to the normal visual applica?on state
-‐ an analy?c descrip?on of the interac?on / naviga?on possibili?es
• Two sogware applica?ons were developed and u?lized
-‐ one for measuring the spa?aliza?on accuracy for each set of earcons
-‐ one for assessing the user ability to imprint a given grid state from a specific auditory display state
Results -‐ Level #1 test applica?ons
!
• Measuring the spa?aliza?on accuracy
-‐ buVons for playing back the earcons sounds on demand and proceeding to the next sample,
-‐ nine buVons corresponding to the ?c-‐tac-‐toe nine grid-‐cells posi?ons
-‐ the users were prompted to select the perceived posi?on for each sample before proceeding to the next one
Results -‐ Level #1 test applica?ons (cont’d)
!
• Inves?ga?ng the user ability to imprint a given grid state
-‐ A significant “metric” associated with the playability of the game in terms of the earcons’ design followed
-‐ Users were requested to select a state, namely “X”, “O” or “empty” for each grid posi?on following the scenario played back
-‐ 5 tes?ng scenarios:
‣ 1 -‐ 3 mainly for occupied posi?ons (“X” or “O”)
‣ 4 -‐5 mainly for the “empty” earcons
Another significant issue tracked prior to the systematic evaluation of the overall sonic design and game-play was the fact that the rolling pellet movement should be limited to the game grid dimensions. If this restriction were not applied, a constant rotation of the grid flat surface would cause the pellet to roll outside the grid physical dimensions. Hence, the concept of the “rolling-stop” virtual wall was introduced, and a smoothed, crashing auditory icon was created. This audio sample was again processed using binaural technology, aiming to spatially locate it towards the four grid edges (see Figure 3).
4. ASSESMENT AND RESULTS In order to assess the efficiency of the tic-ta-toe audio – game prototype realization, we have organized a two-level subjective test. The first level considered mainly the sonic design efficiency, taking into account the particular requirements imposed by the game scenario and rules. In these tests, 20 non-audio expert adult subjects participated. Specifically, the subjects were first given a demonstration of the system, including a detailed explanation of the correlation of the different earcons to the normal visual application state and an analytic description of the interaction / navigation possibilities. At the second test level, this non-interactive demonstration session was followed by a limited five-minute period during which the subjects were allowed to play several sets of tic-tac-toe.
In order to investigate our earcons design efficiency during the first test level, two additional software applications were developed and utilized. The first one aimed to measure the spatialization accuracy for each set of earcons designed. Each subject has used a graphical interface shown in Figure 5, providing buttons for playing back the sound, proceeding to the next sample, as well as nine buttons corresponding to the tic-tac-toe nine grid-cells positions. Each sample could be repeatedly played back, however once the “Next” button was pressed, the user could not repeat the test for the previous sound. Moreover, the user was prompted to select the perceived position for each sample before proceeding to the next one. Each audible sample corresponding to a different cell state or action (i.e. “X” or “O” mark placement) was presented twice, in a fully randomized order.
Figure 5. Application interface, reception of earcon
localization
The second test application was designed to examine the user ability to imprint a given grid state from a specific auditory display state. This is a very significant assessment, since it defines the playability of the game in terms of the earcons’ design
followed. During the tests, the human subjects used an interface as shown in Figure 6, similar to the previous one. The user in this case was requested to select a state, namely “X”, “O” or “empty” for each grid position following the scenario played back. Five testing scenarios were created: Scenarios one to three consist of the earcons for occupied positions (“X” or “O”), while the scenarios four and five moreover include the “empty” earcons. The three first scenarios present respectively four, two and six occupied grid cells. The scenarios’ description is summed in Table 1.
Figure 6. Application interface, reception of auditory scenario
Figure 7 summarizes the results obtained from the first testing application. The percentage shown represents the correct grid placement for the “empty”, “O” and “X” group of earcons. While slightly different earcons are used as a confirmation of a placement action and information for an occupied block, these are concatenated under their respective symbol. Figure 8 also illustrates the results obtained from the second testing application. The red set of columns shows the percentage of the audio scenarios that have been accurately mapped, while the blue set of columns show the percentage of correct grid placement per block within each scenario.
Table 1. Test scenario summary
Scenario “X” Occurrences
“O” Occurrences
“Empty”
Earcons
1 2 2 NO
2 1 1 NO
3 3 3 NO
4 2 2 YES
5 1 1 YES
From these test results, the following conclusions can be briefly drawn:
(a) Less than 30% of the “empty” earcons are correctly placed; however, more than 70% of the “O” and “X” cues are accurately positioned, indicating the strong association between the actual earcon design and their spatial perception.
(b) The results obtained for scenarios 1 and 2 differ minimally compared to the results for scenarios 4 and 5, respectively: Therefore, the presence of “empty”
Results -‐ Level #1 tests outcome
! !
Average correct placement for the earcons employed, nine posi?ons per set
Average correct placement for the five test scenarios
Results -‐ Level #1 tests outcome (cont’d)
• Less than 30% of the “empty” earcons are correctly placed
-‐ however, more than 70% of the “O” and “X” cues are accurately posi?oned, indica?ng the strong associa?on between the actual earcon design and their spa?al percep?on
• Scenarios 1 and 2 results differ minimally compared to the results for scenarios 4 and 5, respec?vely
-‐ the presence of “empty” earcons does not seem to affect the subject mapping capability
Results -‐ Level #1 tests outcome (cont’d)
• Simple auditory display scenarios, including a small number of occupied blocks are easily mapped, with an accuracy percentage up to 80%.
• The minimum percentage of accuracy for
-‐ the auditory display mapping on a scenario basis is 45%
-‐ on a grid block basis the minimum is 75%
-‐ Thus, a limited number of erroneous choices lead to mapped scenarios being dismissed
Results -‐ Level #2 tests
• the users were allowed to play mul?ple ?c-‐tac-‐toe sessions within a limited ?me interval.
• At the beginning this was a rela?vely difficult task
-‐ it mainly resulted into random user ac?ons and symbol placements on the ?c-‐ta-‐toe grid area
• It finally turned out that ager a maximum of three repe??ons, the game-‐play was natural and feasible
• All subjects responded posi?vely to the ques?on regarding the ease of game-‐control through the auditory rolling pellet mechanism
• They also verified that the proposed earcons sonic design conceptually fits to the employed naviga?on mechanism
Conclusions
• We demonstrate and assess a ?c-‐tac-‐toe game adap?on to an audio-‐only environment
• Implementa?on suitable to be supported by any mobile pla[orm equipped by specific movement-‐tracking accelerometer and gyroscopic sensors
• Earcons are employed as the fundamental means of sonifying the informa?on required to construct the necessary auditory display
Conclusions (cont’d)
• Earcons’ design overview
-‐ was an itera?ve process, leading to a robust and efficient set of spa?alized earcons.
-‐ although earcons concurrent / parallel representa?on was not required, however, the spa?al characteris?cs of the sonic mo?ves enhanced the degree of user immersion.
-‐ the final sonic design considered the user control mechanism developed and employed, providing an integrated, mul?modal interface for playing the game
Conclusions (cont’d)
• The efficiency of the audio-‐game prototype was assessed following a sequence of subjec?ve tests
• Most subjects pointed out that
-‐ considerable effort was required to confirm the possible posi?on selec?on through the relevant audio playback, a fact also confirmed by the assessment results
-‐ considerable skill was necessary in order to visualize the game state and provide the next input choice; obviously, this “complexity” increases through each game step
-‐ while the discrimina?on between earcons for “X” and “O” was apparent, some found it difficult to specifically iden?fy the “X” and “O” symbols during the tes?ng phase
-‐ the game play is natural and enjoyable, requiring though focused aVen?on
Future work
• Further playability tes?ng
-‐ track intended and actual game inputs
-‐ correlate the ?me required for visual and audio game play, as a metric of the possible effort required
• Tests by visually impaired subjects are expected to substan?ally differen?ate the assessment results, given their increased abili?es to visualize the game grid and ac?ons
• User centric adapta?on: allow users to define their own earcons mappings from a pre-‐defined, limited set
