Multimodality

Multimodality: How many senses do we have?Alistair D N EdwardsDepartment of Computer Sciencehttp://www-module.cs.york.ac.uk/advt/classes/class01/materials/Multimodality.pptx

1Multi-XMULTI-media, MULTI-modal, MULTI-channel... there are many terms that can be prefixed by the qualifier 'multi', so that there is a danger of getting diverted into debates about the meanings of the terms and distracted from the real questions. At the same time, these discussions often centre on computers; a linguist or a playwright or a singer would not even use the terms, far less wonder whether what they do should be prefixed by 'multi'.How many senses do we have?FiveExtra senseshttp://www.bbc.co.uk/radio4/science/extrasenses.shtml1Pain2Balance3Time4Temperature5Digestion Class of 20159 Body position10 Heat11 Danger12 Circadian Rhythms13 Hunger14 Bodily needs15 Time

16 Pain17 Fatigue

How many senses do we have?Previous answers1 Sight2 Smell3 Taste4 Hearing5 Touch6 Balance7 Direction8 Humour9 Body position10 Heat

11 Danger12 Circadian Rhythms13 Hunger14 Bodily needs15 Time/Rhythm16 Pain17 Fatigue18 Pressure19 Empathy

20 Intuition21 Fear22 Hunger23 Tiredness24 Thirst

What is a sense?What is a human sense?(Lets not get distracted by questions as to whether animals or even plants have senses in the way we mean)What is a human sense?An input channel?Only for external information?or internal too?

What is a human sense?An input channel?Only for external information?or internal too?

SensesExternalInternalCompound?A synthesis of inputsfearhumourempathy?ObjectivesWhat modes do we use in interaction?Which others could we use?Why?How?Individual sensesCombining senses/modesSome practical(?) possibilitiesSoundsHapticsSmell

SoundsSpeechsynthesizedNon-speechearconsauditory iconssonificationdata mappingaudificationprograms

Synthesized speechWho is this?and this?Which is the better voice?How can we make synthetic voices more acceptable?Better use of pauses?Newell, C (2009) Place, authenticity, and time: a framework for liveness in synthetic speech University of York PhD ThesisNewell, C., Edwards, A. D. N. & Cairns, P. (2011) 'Liveness' in human-machine interaction, International Journal of Performance Arts and Digital Media, 7, 2, pp.221-237.

EarconsSymbolic sounds with arbitrary mappings to their meanings

Paint

Open

Open paintEarconsSteve Brewster, Glasgow

Brewster, S.A., Wright, P.C. & Edwards, A.D.N. (1992). A detailed investigation into the effectiveness of earcons. In G. Kramer (Ed.), Auditory display, sonification, audification and auditory interfaces. The Proceedings of the First International Conference on Auditory Display, Santa Fe Institute, Santa Fe, NM: Addison-Wesley, pp. 471-498

Brewster, S. A (1991). Providing a Model for the Use of Sound in User Interfaces, University of York DPhil thesisAlgebra earcons

Algebra earconsStevens, R.D., Brewster, S.A., Wright, P.C. & Edwards, A.D.N. (1994). Providing an audio glance at algebra for blind readers. In G. Kramer & S. Smith (Eds.), Proceedings of ICAD'94, Santa Fe Institute, Santa Fe: Addison-Wesley, pp. 21-30Stevens, R. D. (1996) Principles for the design of auditory interfaces to present complex information to blind people, University of York DPhil thesis

Auditory iconA cartoon sound that resembles the thing it representsSonic finder

Auditory iconGaver, W. W. (1989). The SonicFinder: An interface that uses auditory icons. Human-Computer Interaction, 4(1), 67-94Spearcons

Somewhere between speech and an earconspeeded up speechSpearcon 1Spearcon 2Spearcon 3

SpearconsWalker, B. N., & Kogan, A. (2009). Spearcons enhance performance and preference for auditory menus on a mobile phone. Invited paper in Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction (UAHCI) at HCI International 2009, San Diego, CA, USA, (19-24 July).Sonification

SonificationEdwards, A. D. N., Hines G. and Hunt, A. (2008). Segmentation of biological cell images for sonification. 2008 International Congress on Image and Signal Processing (CISP 2008), Sanya, China, IEEE Computer Society, pp. 128-132.SoundsWhy do we not make more use of sounds in interfaces?Annoyance?What makes a sound annoying?Hapticshttp://www.dcs.gla.ac.uk/~stephen/papers/CHI2000.pdf TermDefinitionProprioceptiveRelating to sensory information about the state of the body (including cutaneous, kinesthetic, and vestibular sensations).HapticRelating to the sense of touch.VestibularPertaining to the perception of head position, acceleration, and decelerationKinesthetic

Meaning the feeling of motion. Relating to sensations originating in muscles, tendons and jointsCutaneousPertaining to the skin itself or the skin as a sense organ. Includes sensation of pressure, temperature, and pain.TactilePertaining to the cutaneous sense but more specifically the sensation of pressure rather than temperature or pain.Force FeedbackRelating to the mechanical production of information sensed by the human kinesthetic system.Haptics - definitionsGeomagic Touchhttps://youtu.be/nmrviXro65ghttp://www.sensable.com/

Formerly the Sensable Phantom OmniInvisible 3-D hapticshttp://phys.org/news/2014-12-team-invisible-d-haptic-video.html

Invisible 3-D hapticsCarter, T., Seah, S. A., Long, B., Drinkwater, B., & Subramanian, S. (2013, October). Ultrahaptics: multi-point mid-air haptic feedback for touch surfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology (pp. 505-514). ACM.OlfactoryTrisenx Scent Domehttp://www.trisenx.com/

iSmell

iSmell

Olfactory deviceodeOlder people who may forget to eatReleases food smells at meal times

odeFragrancesFresh Orange JuiceCherry Bakewell TartHomemade CurryPink GrapefruitBraised Beef CasseroleBlack Forest Gateau

SmellsWhy do we not make more use of smells in interfaces?Combining sensesCombining sensesThe bandwidth problem

VisionHow much information do we receive visually?The real power is in being able to focusVision versus auditionHow many screen objects could we hear?~ 8Pitt, I. J. and Edwards, A. D. N. (1995). Pointing in an auditory interface for blind users. in Intelligent Systems for the 21st Century: Proceedings of the 1995 IEEE International Conference on Systems, Man and Cybernetics, (Vancouver), IEEE. pp. 280-285.Can use auditory streamsBregman, A, (1990) Auditory Scene Analysis, MIT PressMitsopoulos, E. N. and Edwards, A. D. N. (1999). A principled design methodology for auditory interaction. in M. A. Sasse and C. Johnson (eds.), Proceedings of Interact 99, (Edinburgh), IOS Press. pp. 263-271Combining sensesCan the whole be greater than the sum of the parts?T4: turning talking tactile tabletTouchSpeechNon-speechRotation haptics(Vision)https://youtu.be/4nW9-zKmKUw

Conflicting sensesMcGurk Effect

https://youtu.be/G-lN8vWm3m0

Visual dominancePsychologyNeurologyUser interfacesConflicting sensesCrispsZampini, M., & Spence, C. (2004). The role of auditory cues in modulating the perceived crispness and staleness of potato chips. Journal of Sensory Science, 19, 347--363

ConclusionsWhy do we not make more use of other senses/channels/modes in interaction?Which ones could we practically use more?What would be the benefits?Visual veracity

Visual veracity