Sensory Channels and Media Systems: Vision, …fisher/SG03CourseIntersensory.pdfTHE UNIVERSITY OF...

THE UNIVERSITY OF BRITISH COLUMBIA

Sensory Channels and Media Systems: Vision, Hearing, & Haptics

Brian Fisher

Media and Graphics Interdisciplinary Centre

Fisher: Sensory Channels

Intersensory Interactions

• Intro and metacognitive gap

• Integrating cogsci theory with design

• Cognitive Architecture

– Modularity and multimodal interaction

• Information hiding-- conflict resolution

• Cognitive impenetrability

• Performance differences between modules

• Recalibration

– Spatial indexes in complex environments• Multimodal cue matching within modules


Vision systems to multimodality

• Ron: Vision systems and subsystems– Pre-attentive vision (gist, layout, events)

– Attention (grab ~5 objects for processing)

– Combine for “virtual representation”

• Extend system concept to modalities– Some are similar across modalities

– Some are multimodal

– Some are task-dependent


Key Points for HCI Practice

• The Metacognitive Gap: The need for a grounded Cognitive Science approach

• Reflective design practice methods– Integrating CogSci with interaction design– Iterative design cycle

• Examples of extended HCI – Cognitive architecture: Multimodal displays and

how they are understood – Situated cognition: embodied interaction with

complex displays


Introduction

History -- why we need to adapt HCI methods for multimodal interaction


HCI History• “Classic” HCI: User/Task/Tool Model

– Task, Protocol & GOMS/keystroke analysis

• Command-line, menus, workplace systems

• Theoretical underpinnings

– Cogsci of conscious thought

– Learning, Memory, Reasoning

– Sequences of operations


Ergonomic HCI domain limits

TaskTool

User

What if I am doing this for fun?What if I want new insights?What if I want to communicate with someone?What if I am exploring a complex environment?


Evolution towards multimodal displays

• WIMP interface: visual semantics– Metaphorical tool icons on desktop

– Direct manipulation

• Information visualization: Information processing in the visual system– Visual analogs of information

– Spatial Instruments


Current development cycle

Walkthrough or experiment

Implement prototype

Design Assess

Craft model does not scale to large design space

Craft


Future challenges

• Perception, cognition, and action in an immersive

multimodal environment populated with objects

events and actors

• Applications in entertainment, cognition,

communication

• Blend of virtual and real spaces… with seams

– Are the rules consistent?

– Can users shift between them?

– Can frames support rule shifts?


Opportunities for creative design

• Environments: Affordances for exploration– Spatial cognition, human space constancy theory

• Support for creative & logical thinking – Problem solving, embodied cognition models

• Media-based communication & collaboration – Metacognition, distributed cognition

• Experience (Kansei) engineering: Moving beyond usability


Effective Interface Design for Rich Sensory Environments

The interaction between display characteristics and the information processing characteristics of the

user’s perceptual, motor, and cognitive processes will largely determine interface performance


Metacognitive Gap

• Intuitions about thoughts,goals and plans (“folk Psychology”) are reasonably accurate

• Intuitions about how people see, hear, and remember are very inaccurate

• Lack of awareness of the limits of intuition is the “Metacognitive gap”

• Design-by-intuition leads to bad user experience


Evolving interaction design models

• Guidelines are (still) inadequate• User-centred design inadequate for

rich sensory environments• Need for theory-rich, evidence-based

approach: design for lower-level processes

• Must integrate with higher-order cognitive task analysis and user-centred design practices


Bridging the theory/practice gap

• Convincing designers that there is something to

understand—the “metacognitive gap” of folk

Psychology

• Convincing Cognitive Scientists to answer relevant

questions—Complex data displays and multiple

tasks and the Psychophysics reductionist approach

• Integrating research and design—Finding a

common language


Multimodal development cycle

Constrain large multimodal design space

Assess specific

aspects of interaction

Literature Foraging:

HCI & graphicsPsychologyKinesiologySociology

Architecture…

Design for key sensory & motor

systems

Walkthrough or experiment

Implement prototype


Fields of interest: Experimental Psychology:

– Founded~ 100 years ago

– Areas of study• Psychophysics—Vision, hearing, tactile senses

• Attention—Endogenous, exogenous, sustained

• Learning and memory

• Goal is often information processing algorithms

• Discussed in Module 2


Useful things from Psychology

• Perception and attention– Visual & auditory acuity & discrimination

– Colour perception

– Salient display changes, change blindness

• Learning and memory– Primacy, recency

– Skill acquisition

– STM limits

– State-dependent learning

• Decision making– Base-rate neglect

– End effects, biases


Sound perception basics

• Auditory Psychophysics – Waves of air pressure, 16Hz - 20kHz.

– Sensitivity fits Stevens Power Law

– Frequency masking (used in compression)

• Hearing in a complex world– Auditory “streams”

– Auditory localization

– Acoustic communication


Complex auditory scenes • Auditory stream segregation system

(Bregman)• Gestalt grouping rules

– Proximity:

– Similarity:

– …but in frequency and timbre spaces

http://tinyurl.com/i2hf

XXX XXX XXX

X X X O O O X X X


Acoustic communication• Speech

– Highly rule-based

– Categorical perception

– Multimodal speech (visemes)

• Music– Somewhat rule-based

– Stream segregation

• Soundscape– Some regularity “Acoustic ecology”

– Spatial & temporal coherence w. vision, haptic

– Sound events lead us to search for their cause


Spatial sound

• Left vs. right (ear)

– Relative intensity (high freq.)

– Onset time difference (high freq., rapid onset)

– Phase differences (low freq.)

• Azimuth, distance

– Room acoustics

– Pinna reflections/attenuations (HRTF)

• Spatial sound is evocative (Audium: Shaff)


Fields of interest: Kinesiology & related disciplines

• Science of human movement – Neuroscience

– Mechanics

– Anthropometry

• Examples: Fitts’ law, GOMS/Keystroke

• Goal is often perceptually guided behaviour


Haptic perception basics

• Cutaneous, tactile & somatic system

– Stimulus to the skin

– Heat, pressure, vibration, slip, pain

• Kinesthetic & proprioceptive system

– Limb position, motion, force

– Receptors in muscles, tendons, and joints

– Stimulated by bodily movements


Cutaneous, tactile & somatic system

• Thermoreceptors: change in skin temp

• Mechanoreceptors: pressure, vibration, slip

• Nocioreceptors: pain

Systems within modalities: (similar structure to vision, audition)

But we still have access to some haptic “primitives”


Haptic bandwidth of finger & hand

5000-10000 Hz : Sense vibration in manipulative task

320 Hz : Discriminate two consecutive force input signals

20-30 Hz : Min. force input for meaningful perception

12- 16 Hz : Can correct grasping forces if the object slips

8 - 12 Hz : Can correct for positional disturbances

5 - 10 Hz : Max comfortable force & motion commands

1 - 2 Hz: Max to react to unexpected force/position signal

Shimoga,1992


Distribution of tactile sensors


Kinesthesia

• Perception of limb position, motion, force

• Main information from mechanoreceptors

– “Force sensors”: golgi tendon organsMeasure tension between muscles & tendons

– “Position sensors”: muscle spindlesLocated between individual muscle fibersexcited by changes in muscle length (stretch)


Fields of interest: Cognitive Science

• Cogsci Society founded 22 years ago• Combines Experimental Psychology, AI,

Philosophy and Neurophysiology • 3 levels of analysis

– Semantics: Intentions, Goals, and Meanings– Syntax: Information processing– Implementation: Neural processing

• Goal is often Cognitive Architecture


Other fields of interest

• Psycholinguistics

• Social cognition

• Cultural Psychology

• Communication theory

• Embodied communication, paralinguistics

• Anthropology


Example of metacognitive gap

• The “computer metaphor” of mind

• Intuition: Single mental processor reads all

senses and performs a variety of tasks.

• Cognitive Science: Processing modules

operate in parallel.

– Restricted flow of information and control.

– Processing characteristics are counterintuitive


“Horizontal” Modularity

• Model Human Processor (MHP)

• Serial stages of processing

• Information flow is Bottom-up

– Cognitive impenetrability

– “Seeing is believing”

Action

Cognition

Perception


Horizontal modularity restrictions

Cognitive impenetrability (Pylyshyn, 1984) refers to the inability of observers to use semantic information (such as what the person believes or intends to do) to influence the operation of the input stage.


Vertical modularity (Fodor)

Phoneme perception

Voice recognition

Auditory localization

Cognitive processing


Vertical modularity restrictions

Information encapsulation (Fodor, 1983) refers to structural barriers within the

cognitive architecture that prevents internal data stores from being shared between

modules in the same stage of processing.


Events are processed by many channels

• Intuitions about thinking

– Fails at low levels

• Cognitive Architecture

– Multiple brain areas

– Interconnected

– Informationally encapsulated

– Multimodal inputs parsed from scene

Cognition

DorsalSystem

Bimodalspeech

Ventralsystem

…

Sensory world


Cognitive Impenetrability & Modularity

• Stroop effect– Reading is data-driven module– Competition for response

• Other Modularity phenomena– Modularity of perception for action– Modularity of visual/auditory integration– Modularity of eye movement control– Modularity for Models of Minds


Advantages of modular processing

• Download task to module, reduce cognitive bottlenecks

• Fast, effortless information processing

• Near-optimal information integration between cues and sensory modalities– Fuzzy logic cue integration

– Bayesian categorization


"Civilization advances by extending the number of important operations which we can perform without thinking about them."

Alfred North Whitehead

What are the disadvantages?


Disadvantage: Processing rigidity

• Stress within a module is not accessible• Complex stimuli may be processed

differently in different modules• Tasks may access different modules with

different performance characteristics• Virtual environments may introduce

discrepancies that impact different modules (and tasks) differently


Stress within a module may be undetected

Poor cognitive access to low-level processes

Action

Cognition

Perception


Example: VDT stress syndrome

• Users complain of headache, vision problems etc.

• Reports anecdotal, but reading impairment is observed

• Also pupillary tremor and regressive saccades


Reflective HCI Practice (after Schön)

Test: walkthrough, field study or experiment

Exp:“Lab sense”,FS: Observation

Ideas and hypotheses

Evaluation, Mapping

Implement

Interaction Design

Information foraging

(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…


Reflective HCI Practice: Foraging






Theory of space constancy in active vision

• Cognition needs the illusion of a stable world

• Eye movements should create confusing image shifts

• Maintaining space constancy requires– Efference copy

– Passive blur

– Lateral masking

– Saccadic suppression


Reflective HCI Practice: Hypothesizing




Evaluation, Mapping




VDT fatigue study

• Hypothesis “Sampling” raster during saccades reduces intra-saccadic blur, and may overload saccadic suppression

• Space constancy perspective allowed us to:– Isolate the important factors in a complex

situation

– Find a more sensitive task and measure

• Study examines detection of movement during saccade (w Bridgeman & Macnik)


VDT fatigue study recommendations

• Suppression thresholds elevated for flickering stimuli

• Effect is reduced >250 Hz

• Problems will be greatest – large saccades

– high-contrast display

• Work arounds include blanking display during saccades


Reflective HCI Practice: Testing




Evaluation, Mapping

Implement

Interaction Design




Moving to multimodality

Vision

Virtualinteraction model

Force & tactilefeedback

Psychophysics of vision, sound, and touch will change when environment is multimodal

Hearing


Extending to complex worlds

• Lab studies: few events, visual or auditory

• In contrast to multimodal interfaces

– Virtual worlds

– Augmented reality

– Ubiquitous computing

• How are multiple multimodal events dealt

with in the brain?


Some systems are multimodal

Example: Cross-modal speech system

• Reduces cognitive load

– Fast, effortless information processing

• Near-optimal information integration between cues and sensory modalities

– Fuzzy logic cue integration

– Bayesian categorization


Cross-modal speech perception

The McGurk Effect: McGurk & MacDonald 1976


Illusory conjunctions occur in artificial multimodal environments

Example: Movie theatre

• The McGurk effect (face influences sound)

– Dubbed movie

• The ventriloquist effect (vision captures sound location)

– Sound seems to come from actor


Rensink


Study: Pointing to sounds • Cognitive location good

• Pointing shows visual capture

– Aware of visual and auditory locations, but point to visual

• No effect of phoneme/viseme fit on ventriloquism

• Slow recalibration of auditory space if offset constant

“Ba”

What was sound?Where was source?(point or describe)


Our interpretation: 2 systems at work

• Different multimodal systems solve feature

assignment problem differently

– Motor system: high visual dominance

– Cognitive system: low visual dominance

– Phoneme/viseme mismatch doesn’t help

• Vision can recalibrate spatial sound “map”


Rensink


Attentional pointers in systems?


Action (motor space)


Visual localization


Study: Pointing in large displays (Po)

• Tell me where the target is

• Point with no feedback

• Point with visual feedback (cursor)

• Point with delayed visual feedback


Findings

1. Can you tell if a target is on the left or right?

• 3 out of 7 males, 7 out of 7 females made errors

2. Can you point to it with no visual feedback?

• 6 out of 10 who failed #1 were correct

3. Are you better with a (simulated) laser pointer?

• Out of 6 who point accurately in 2, all fail

4. Will pointing accuracy be affected if visible pointer lags

pointing?

• 3 of the 6 who failed #3 succeed

All results predicted by 2 visual systems

hypothesis


Displays and multimodal perception

• Immersive environments must display a complex multimodal world– Virtual Reality must provide entire world– Augmented Reality must blend with real world

• Multimodal displays have errors– Location of events is not precise (esp. in depth)– Timing is not precise– Graphics can be low-fidelity

• What will be the impact of these errors on users?


Disadvantage: Each module must solve feature assignment problem.

• Modules can’t accept information from other modules: Information encapsulation

• Different modules should have access to a different set of matching cues.

• Illusory conjunctions can occur in multimodal environments:– Phoneme perception: The McGurk effect

– Auditory localization: The ventriloquist effect


Study: Impact of display errors on multimodal perception

• Immersive environments typically have display errors– Location of events is not precise– Timing is not precise– Graphics can be low-fidelity

• As immersive environments add sound and touch, what will be the impact of these errors?


Reflective HCI Practice:Foraging






Recalibration by pairing (Epstein, 1975)

• Individual senses adapt to display • Sensory modalities calibrate each other:

haptics, vision, sound– Observed actions calibrate visual space (space

constancy) – Vision calibrates hearing for the location of a

multimodal event– Sound calibrates vision for the time of a

multimodal event

• Result is an after-effect: remapping of auditory (visual, haptic) space


Reflective HCI Practice: Hypothesizing




Evaluation, Mapping




Impact of information encapsulation

• Multimodal environment with errors in timing and location

• The same event might give rise to a single multimodal construct in one task, and two unimodal events for another.– Vary location of visual and auditory phonemes

in a simple teleconferencing-style video display

– Vary information carried by using synthetic speech stimuli (5 levels).


Ventriloquism meets the McGurk effect.

• Vary location of visual and auditory phonemes in a simple teleconferencing-style video display

• Vary information carried by using synthetic speech stimuli (5 levels).

• Subjects report sound location and syllable heard,

• Analyses included testing a variety of mathematical models of information integration by fitting free parameters with STEPIT.


Use of mathematical modeling tools allow us to address

• Sensory input from a number of channels simultaneously

• How stimuli from multiple channels are matched and partitioned into mental representations

• How information from multiple senses is integrated to give rise to trans-modal mental events


Reflective HCI Practice: Testing




Evaluation, Mapping

Implement

Interaction Design




Results:

• Visual capture of auditory source location, resulting in a shifting of unimodal auditory location estimation (ventriloquism after-effect).

• No effect of location difference on phoneme perception as measured by statistical or modeling tests.

• No correlation between errors in the two tasks (i.e. subjects could not selectively attend to the auditory phoneme on trials when visual capture failed).

• Overall, modularity of phoneme perception is supported.


Disadvantage: Changes in task interact with modules to change performance

• 2 visual systems—“ventral stream” for recognition and “dorsal stream” for action.

• Where vs how• Different impact of illusions• Lesion data


Functional Neuroanatomy of perception for action.

2 visual systems—“ventral stream” for cognition and “dorsal stream” for motor performance.


2 visual systems lesion evidence

lesion performance deficits spared abilities

V1 (blindsight) detection and identification

pointing

Ventrolateral occipital (DF)

identification, shape recognition, object orientation

object manipulation (orientation matching, grip scaling)

Posterior parietal (RV)

object manipulation (orientation matching, grip scaling)

identification, shape recognition, object orientation


2 visual system illusions

stimuli deficits spared abilitiesTichner circles size report grip scaling

displacement during saccade

detection of displacement, location report

pointing

Moving or off-centre frame

induced motion, location report

pointing


Research with videoconferencing and abstract displays

• Targeting sound: cognitive better than motor – Subs aware of visual and auditory

locations, but point to visual

• Targeting vision with context: Less feedback is better– Pointing with no visual feedback better

– Lagged cursor better than unlagged


Interpreting pointing studies

• Pointing studies counterintuitive, but

predicted by response characteristics

of neurons in dorsal/ventral to visual

and auditory stimuli


2 visual system illusions

stimuli deficits spared abilitiesTichner circles size report grip scaling

displacement during saccade

detection of displacement, location report

pointing

Moving or off-centre frame

induced motion, location report

pointing

Sound with displaced visual distractor

pointing apparent location of sound


Modularity Recap

• Displays can interact with active perception to cause hidden stress: VDT study

• Displays can impact matching cues for multimodal cue integration– Support or frustrate recalibration by pairing– Lead to discrepancies in number and

composition if stimuli in different modules• Displays can cause disagreement between

motor performance and cognitive measures• Sometimes removing information helps


Extending to complex worlds

• Previous studies in simple worlds, with a few visual and auditory events

• Multimodal environments are complex– Virtual worlds

– Augmented reality

– Ubiquitous computing

• How are multiple multimodal events dealt with in the cognitive architecture?


Indexical cognition (Pylyshyn)


“FINSTs... make thoughts true”

• Perception– “Hotlink” tokens– Drawn to salient events– Object-centred, “sticky”– Visual routines– Finite number ~ 4

• Cognition– Maintain object history– Implicit memory of object associations– Sparse cognitive representation– Just-in-time delivery of information– Atom of intentionality


Mental representations of complex worlds

• Cognitive architecture perspective requires that links be established between lower level perceptual qualities and cognitive symbols—i.e. a pointer, called a FINST.

• FINSTing allows us to interact with perceptual objects and events without the need for mental images per se.

• Symbolic representation + pointers makes different predictions than intuitive picture-in-the-head


Indexical cognition (Pylyshyn)


Naïve view of FINSTs in Cognitive Arch

Phoneme perception

Voice recognition




FINSTs


Another view of FINSTs

Phoneme perception

Voice recognition




FINSTs


More about FINSTs

• FINSTs Link mind & perceptual world– Visual routines: (collinear, inside,

subitizing)– History of an object– Object-centred, “sticky”– Drawn to salient changes-- onsets,

luminance increments, oddballs– Finite number ~ 4-7– FINSTs + ANCHORs for motor

behaviour


More about ANCHORs

• ANCHORs link mind & action– Remembered locations for eye

movements

– Direct interaction with items off the retina

– Fast, robust motor performance by action routines

– Affordances for action


Role of haptics

• See “physical interaction:human haptics” by Karon MacLean


Multiple object tracking demo


Another trial (Scholl)


Another trial (Scholl)


Application


Mental representations of complex environments

• Cognitive architecture perspective requires that links be established between lower level perceptual qualities and cognitive symbols—i.e. a pointer, called a FINST.

• FINSTing allows us to interact with perceptual objects and events without the need for mental images per se.

• Symbolic representation + pointers makes different predictions than intuitive picture-in-the-head

• Coping with spatial transformations in complex data spaces


Multimodal representations are virtual

All modalities store little info in memory:

instead they take up information as needed

– Vision-- attention, eye, head and body

movements change view

– Haptics-- active exploration of space with hands

– Hearing-- uses body and head movements to

localize sound and improve quality


Multimodal events support adaptation

• Individual senses adapt to display

• Modalities use multimodal events for cross-calibration– Observed actions calibrate visual space

– Vision calibrates sound location

– Sound calibrates vision for time

• Result includes after-effect: a remapping of perceptual space

(Epstein, 1975)


Research question: Role of focal attention?

Are attentional resources shared between senses?

• Will adding sound and haptics impact visual attention?– Or, will it offload processing from vision?

• Does a shift in one modality cause complementary attention shifts in others?

• Does recalibration require attention?


Research Topic: Pointers for action?

• Attentional pointers link mind and world

• Do “action pointers” link mind & muscles?– Remembered locations for eye movements

– Direct interaction with items off the retina

– Fast, robust motor performance by action routines


Research Topic: Individual differences

• Perceptual rules are the same

• Impact differ over time and for individuals

– e.g. sensitivity to stereo depth & spatial sound cues

– Ability to adapt to new cue combinations

• Perceptual customization may help

– For individuals: “personal equation” for interaction

– In real time, through attentive computing


Module advantages

• High-realism interface designs improve

performance by “downloading” information

processing operations to input modules.

• Interaction of display characteristics with

capabilities and characteristics of the

functional architecture will determine

performance.


Module disadvantages

• Coordination– Distortions in location, timing, and category-relevant

information may lead to the formation of conflicting representations in different modules.

• Processing inflexibility– Errors and conflicts within a module can create errors and

increase cognitive load. (CRT flicker example)

• Information hidingCognitive impenetrability of modules makes it difficult for

operators to determine the reasons for their poor performance.


Inseparability of Mind & World

• Embodied cognition-- mind/body

• Situated cognition-- mind/world

• Distributed cognition-- mind/mind

• Ecological theories (Vygotski, Luria, Bateson, Gibson) can be linked to sensory phenomena and inform interaction design


Key Points for HCI Practice

• The Metacognitive Gap: The need for a grounded Cognitive Science approach

• Reflective design practice methods– Integrating CogSci with interaction design– Iterative design cycle

• Examples of extended HCI – Cognitive architecture: Multimodal displays and

how they are understood – Situated cognition: embodied interaction with

complex displays


Environment/Inhabitant/Representation

• Information visualization

• Perceptual/deictic/situated Cognition

• Cognitive Architecture, perception, attention

• Display as extension of mental model


What to expect in the next talk

• More on haptics• Other senses

– Neuromuscular,GSR, heart rate, brain, other biopotentials

• Applications– Displays, input, & sensing technologies– Design examples– Virtual environments

• Communicating human experience: information, emotion, environment

– Intimacy and embodiment– Sources of aesthetics

Date post:	14-May-2018
Category:	Documents
Upload:	hadat
View:	217 times
Download:	2 times

Sensory Channels and Media Systems: Vision, …fisher/SG03CourseIntersensory.pdfTHE UNIVERSITY OF...

Documents