Date post: | 12-Jan-2016 |
Category: |
Documents |
Upload: | stephanie-washington |
View: | 215 times |
Download: | 0 times |
Musical Program Auralisation 1
The cAitlIn systemMusical Program Auralisation
& External Representations of Pascal
http://computing.unn.ac.uk/staff/cgpv1/caitlin
Dr Paul VickersSchool of Informatics, Northumbria University
Professor J.L. AltyDean of Faculty of Science, Loughborough
University
Musical Program Auralisation 2
Introduction
Major features Background to the project Sound in HCI A case for music The cAitlIn system Demonstration Future work
Musical Program Auralisation 3
Major features
With cAitlIn we have:– A musical equivalent for various syntactic
components of a programming language = musical grammar
– Extended the earcon technique (q.v.)– An audio program structure diagram
(execution path)
Musical Program Auralisation 4
Major features– A MPAS usable by non-musicians– Shown that music can be used to
communicate information about program state and structure
– In turn, this info can assist bug location
Empirical evaluation Organising principles Opened new line of enquiry &
researchAuditory External Representations
Musical Program Auralisation 5
Background
Interests in music, computing, & programming
Hunch that music could be used to assist in the programming process
Develop and test a musical framework for communication of program information
Musical Program Auralisation 6
Sound in HCI
Increasing research focus since mid ‘80s. A belief that non-speech audio useful as
interaction medium A nascent field called auditory display
spawns techniques like:– Sonification– Audification– Auralisation
Musical Program Auralisation 7
Examples
Auditory icons Earcons Data visualisation (sonification &
audification) Assistive technologies
Musical Program Auralisation 8
Program auralisation
Mapping of program data and events to sound
Main examples– LogoMedia– Sonnet– ADSL– LISTEN
Largely untested
Musical Program Auralisation 9
Why music?
Gives multiple audio streams in parallel
Structured and organised according to specific rules
Time-ordered. Visual techniques are spatial.
Can music’s grammars be used to carry real information?
Musical Program Auralisation 10
cAitlIn system
Computer Audio Interface to Locate Incorrect Nonsense
Uses musical motifs to render Pascal language constructs at run time
Motif design is hierarchic as is organisation of constructs
Musical Program Auralisation 11
Hierarchic motif design
Construct type
Selection Iteration
IFs CASEs
IF...ELSEIF... CASECASE...
ELSE
Determin-ate
FOR
Indeterm-inate
WHILE REPEAT
Abstract auralizationEntry/Body/Exit leitmotifs
Genericselection leitmotif
Genericiteration leitmotif
GenericIF leitmotif
GenericCASE leitmotif
Musical Program Auralisation 12
Musical grammars
Musical Program Auralisation 13
Musical grammars
We map existing grammar (e.g. Pascal syntax, English, etc.) onto a musical grammar
Transliteration or translation? We may get emergent properties
leading to a new musical grammar
Musical Program Auralisation 14
cAitlIn musical grammar
Uses motif classes and variations to render Pascal constructs
Allows– abstraction– nesting (tonality & harmony preserved
across motifs) Uses grammatical rules (e.g. a tonal
music system)
Musical Program Auralisation 15
cAitlIn musical grammar cont Renders information not directly
communicated by source code or other visualisation techniques
a new grammar, conveys extra meaning
Musical Program Auralisation 16
Advantages
Over earcons– do not show state
Over Auditory Icons– symbolic mappings = poor system aural
ecology Syntactic design expect retention Displays both flow and state
Musical Program Auralisation 17
Design of Tunes
Points of Interest– Construct Entry– Evaluation of condition– Execution of body– Exit from construct
Boolean True=major, False=minor Drones
Musical Program Auralisation 18
Selection Tunes
Metaphorically asking & answering a question.
Tune rises on entry and descends on exit
Musical Program Auralisation 19
Iteration Tunes
Major/Minor chord to show when condition evaluated
Heard at start of WHILE loops and at end of REPEAT loops
FOR loops increase/decrease the pitch each time round the loop
Musical Program Auralisation 20
Variations
All selections use same basic tune, modified for individual constructs
All iterations use another basic tune, again modified
Musical Program Auralisation 21
Auditory Parentheses
As () bracket a thought in a sentence, so the auditory parentheses serve to open and close constructs
Musical Program Auralisation 22
Other Statements
All other statements are currently represented by a drum beat, one per statement.
Musical Program Auralisation 23
Demonstration
Musical Program Auralisation 24
Examples
Generic selection motif Generic iteration motif
Musical Program Auralisation 25
IF (true)
a := 10 ;IF a = 10 THEN Writeln (‘A is 10’) ;
Musical Program Auralisation 26
IF (false)
a := 0 ;IF a = 10 THEN Writeln (‘A is 10’) ;
Musical Program Auralisation 27
IF…ELSE (true)
a := 10 ;IF a = 10 THEN Writeln (‘A is 10’)ELSE Writeln (‘A is not 10’) ;
Musical Program Auralisation 28
IF…ELSE (false)
a := 1 ;IF a = 10 THEN Writeln (‘A is 10’)ELSE Writeln (‘A is not 10’) ;
Musical Program Auralisation 29
CASE match
a := 3 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ;END ;
Musical Program Auralisation 30
CASE no match
a := 100 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ;END ;
Musical Program Auralisation 31
CASE…ELSE match
a := 3 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; ELSE Writeln (‘No match’) ;END ;
Musical Program Auralisation 32
CASE…ELSE no match
a := 100 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; ELSE Writeln (‘No match’) ;END ;
Musical Program Auralisation 33
WHILE 2 iters
cntr := 1;WHILE cntr <= 2 DO cntr := cntr + 1 ;
Musical Program Auralisation 34
REPEAT 3 iterations
cntr := 1 ;REPEAT cntr := cntr + 1 ;UNTIL cntr > 3 ;
Musical Program Auralisation 35
FOR…TO 6 iters
FOR cntr := 1 TO 6 DO Writeln (cntr) ;
Musical Program Auralisation 36
FOR…DOWNTO 6 iters
FOR cntr := 6 DOWNTO 1 DO Writeln (cntr) ;
Musical Program Auralisation 37
Sequence & Nesting
Clue to nesting is in the drone If you hear a continuous note then you
are inside a construct. Each construct has a drone. For nested constructs you hear multiple
drones simultaneously Drone pitch increases as nesting
deepens
Musical Program Auralisation 38
Example programProgram ExemplarVAR counter : Integer ;BEGINcounter := 1 ;WHILE counter <= 2 DO BEGIN IF counter MOD 2 = 0 THEN Writeln (‘Counter is even’) ; counter := counter + 1 ; END ;END.
Musical Program Auralisation 39
Scoreiteration open WHILE drone
True
counter := counter + 1;
selection open
FalseIF drone
selection close
WHILE condition
True
IF drone
exit IF
Writeln...sleighbell denotes
terminating conditionfor WHILE
False exit WHILE
iteration close
closed triangle
Auralisation score of aprogram. Notice that
the drones for eachconstruct appear
together on the stafflabelled 'Drones'.
Likewise, all percussiveevents for each
construct appear onthe 'Drums' staff.
Shifts between majorand minor are shown
by changes in keysignature.
1999 Paul VickersAll rights reserved
counter := 1
True
counter := counter + 1;
Musical Program Auralisation 40
Score
Musical Program Auralisation 41
Structure DiagramProgram Sales
Prepare file Process Sales
Sales Record
Valid Sale Invalid Sale
Test sale Read next
Close file
*
o o
Musical Program Auralisation 42
Experiment 1
Listening test to assess comprehensibility of motifs
22 subjects, 60 auralisations Recognition rate better than by
chance and with very little training Hierarchic organisation allowed class
level identification & abstraction
Musical Program Auralisation 43
Experiment 2
Debugging exercise 22 subjects, 8 programs, 4
auralised, 4 normal Results suggest technique is
useful, especially for complex programs and where program output gives few clues
Identified avenues for further research
Musical Program Auralisation 44
Limitations
No comparison with visual task yet Cognitive latency of motifs? No longitudinal results
Musical Program Auralisation 45
Future work
Follow-up studies Extend the grammar to include
assignment statements and procedure calls– How far? Bugs not manifested in program
flow are v. hard to detect and generally require variable inspection to locate, which is impractical for auralisation techniques
Musical Program Auralisation 46
Future work
Port cAitlIn to MS-Windows and conduct a longitudinal study
Assistive technologies. How can MPA aid blind programmers?
Musical Program Auralisation 47
Future work
Preparing EPSRC bid to investigate Auditory External Representations
(c.f. EPSRC CRUSADE project at COGS in Sussex - du Boulay et al)
Musical Program Auralisation 48
Publications–Journals & Reports Vickers, P. and Alty, J. L. (2002). When Bugs Sing,
Interacting with Computers in press. Vickers, P. and Alty, J. L. (2002). Musical Program
Auralisation: A Structured Approach to Motif Design, Interacting with Computers in press.
Vickers, P. and Alty, J. L. (2002). Using Music to Communicate Computing Information, Interacting with Computers in press.
Vickers, P. (2001). Using the World-Wide Web as a Platform for an Interactive Experiment, Liverpool: Liverpool John Moores University, Technical Report CMS-21, ISBN 1-902560-050.
Musical Program Auralisation 49
Publications–Conferences Vickers, P. and Alty, J. L. (2000). Musical Program
Auralisation: Empirical Studies, in Proc. ICAD 2000 Sixth International Conference on Auditory Display (Atlanta, GA, 2-5 April, 2000), International Community for Auditory Display, pp. 157-166 ISBN 0-9670904-1-5
Vickers, P. and Alty, J. L. (1998). Towards some Organising Principles for Musical Program Auralisation, in Proc. ICAD '98 Fifth International Conference on Auditory Display (Glasgow, Nov 1-4, 1998), British Computer Society ISBN 1 902505 05 0
Alty, J. L. and Vickers, P. (1997). The CAITLIN Auralization System: Hierarchical Leitmotif Design as a Clue to Program Comprehension, in Proc. ICAD '97 Fourth International Conference on Auditory Display (Palo Alto, November 3-5, 1997), Xerox PARC, Palo Alto, CA 94304, pp. 89-96
Musical Program Auralisation 50
Publications–Conferences cont. Alty, J. L., Vickers, P. and Rigas, D. (1997). Using Music as
a Communication Medium, in Proc. Refereed Demonstrations, CHI97 Conference on Human Factors in Computing Systems (Atlanta, GA, March 22-27, 1997), ACM Press, pp. 30-31 ISBN 0-8979-926-2
Vickers, P. and Alty, J. L. (1996). CAITLIN: A Musical Program Auralisation Tool to Assist Novice Programmers with Debugging, in Proc. ICAD '96 Third International Conference on Auditory Display (Palo Alto, Nov 4-6, 1996), Xerox PARC, Palo Alto, CA 94304, pp. 17-24