THE ENACTIVE TORCHDr. Tom Froese

IIMAS-UNAM

Traditional interface design

• Cognitivist view of tools:• Subject and tool are

independent• The tool is just another

object in the world• Sensorimotor interaction

loops are ignored• Symbolic information is

given precedence over direct perceptual experience

Froese, McGann, Bigge, Spiers and Seth (2012)

Enactive interface design

• Embodied view of tools:• Subject and tool are

interdependent• The tool is an interface• Interfaces mediate

sensorimotor interaction• Continuous interaction

shapes user experience

Froese et al. (2012)

Paradigmatic example: sensory substitution interfacesTactile-visual substitution system (TVSS)

Tools for studying the embodied mind

Effects of tool-use• Psychologists have been busy documenting the various

ways in which the use of tools affects the users.

• Changes in body schema (living embodiment)• Changes in perceived space• Changes in body image (lived embodiment)

• Usually, but not always, such tool use based changes happen to various extents at the same time.• Effect of modulation, not control.

• But precisely how are these changes constituted?• And precisely what is their main cause?

Tactile-Visual Substitution System (TVSS)

• When blind subjects have been trained to use TVSS for several weeks their experience begins to be transformed.

• “our subjects spontaneously report the external localization of stimuli, in that sensory information seems to come from in front of the camera, rather than from the vibrotactors on their back”

(Bach-y-Rita et al. 1969)

• However, even today there is still little agreement about the best way to interpret this kind of verbal report.

Debating TVSS experience• Prinz (2006): “My best guess is that prosthetic vision devices simple

allow subjects to make automatic inferences about where objects are

located in space as a result of tactile information” [cognitive inference]

• Block (2003): “There is doubt as to whether the phenomenology of

TVSS is exclusively visual” and “perhaps TVSS is a case of spatial

perception via tactile sensation” [tactile perception]

• Noë (2004, p. 27): “It is reasonable to admit that the resulting

experiences are, if not fully visual, then vision-like to some extent”

[visual perception]

• Auvray and Myin (2009): “the experience after sensory substitution is a transformation, extension, or augmentation of our perceptual capacities, rather than being something equivalent or reducible to an already existing sensory modality” [augmented perception]

Methodological issues• Are changes in subject behavior based on

• (a) a change in cognitive judgment based on the same perceptual experience, or

• (b) a transformation in the perceptual experience itself?

• If (b), then is the transformation • Limited to within the existing tactile domain?• Going from the tactile to the existing visual domain?• Going from the tactile to a new, mixed domain?

• How is it possible to distinguish between these options?• We need to return to an investigation of our concretely lived

experience itself using appropriate methods • For a review, see e.g. Froese et al. (2011).

Brecht’s Galileo and the telescope

But gentlemen, why don’t you just try the instrument??

The Enactive Torch

The Haptic Torch (Spiers)

The Enactive Torch

(Froese and Spiers 2007)

Development of technology

Tongue Display Unit BrainPort vision device

The Enactive Torch v2.0

The ET with data capabilities

The “Enactive Torch” (ET) v3.0

SENSOR

MOTOR

Distance-to-Vibration Sensorimotor Substitution

The Enactive Torch v4 & MinET

Comparison between interfaces

TVSS / BrainPort

• Input: light (2D)• stream of black and white

pixel arrays

• Output: touch (2D)• array of mechanical vibration

(back, or stomach), • or array of electrical vibration

(tongue)• Prefabricated representation

• Expensive, intrusive, hard to learn, commercial.

Enactive Torch (ET)

• Input: distance (1D)• a single channel of distance

measures

• Output: touch (1D)• a single motor for mechanical

vibration (hand, wrist, or elsewhere)

• Environment must be enacted

• Cheap, non-intrusive, easy to learn, Creative Commons license.

The ET experimental research platform

• Distance-to-tactile sensory substitution

• Non-intrusive, hand-held lightweight device

• Affordable design and can be easily replicated

• Easy to use and learn

• Allows you to experience how you enact the world

Froese and Spiers (2007)

Technologically mediated coupling

What kind of tool is the ET?• Distal perception? Yes.

• Subjects trained in using this device readily report a transformation in their perceptual experience that involves the appearance of things ‘out there’ in the world. [elongated tool]

• Distal manipulation? No.• In contrast to using elongated tools, this perception at a distance is

not accompanied by a change in the potential for directly acting at a distance. [laser pointer]

• How does using the ET transform our world and body?

Experimental setup• We tested 22 blind-folded participants after training them how

to use the ET in various tasks. • 11 of them were trained more extensively.

• The experimental task was to navigate around a novel maze as many times as possible while limiting collisions.

• We measured both

behavioral responses

(objective data)

and verbal reports

(subjective data) of

using the Enactive Torch.

Summary of experimental procedure

1. Evaluate the participants’ body image of the arm via a tapping and laser pointer procedure.

2. Give a brief introduction to the Enactive Torch and allow participants to briefly get used to it.

3. Ask blind-folded participants to complete the maze as many times as possible while only using device to find their way.

4. Evaluate the participants’ body image again.

5. Evaluate the experience of using the device.

Froese et al. (2012)

Summary of objective results

Froese et al. (2012)

Mann-Whitney U statistical tests did not reveal statistical significances.

Objective performance• Task success is indicative of distal perception:

• Most participants were able to complete the maze around 3-5 times in the given 5 minutes.

• Untrained participants had to learn the precise correlation between distance and vibration by trial and error; yet of overall collisions were relatively few (variation 9-50; median 21 collisions).

• But this purely behavioral evidence does not tell us anything about whether we are in fact dealing with a subjective experience of distal perception.

Aspects of ET user experienceStatement Disagree Neither D. or A. Agree

... I experienced my focus of attention mainly on the obstacles I perceived in front of me.

1 4 16

... I experienced the contours of the maze out in the world at a distance away from my hand.

4 3 15

This seems to support the hypothesis that an ET-mediated sensory-motor loop results in distal perceptual experience.

If tool-based transformations of body image depend on distal perception, we expect measures of perceived arm length to have become extended.

Extended reachability• According to the literature on tool-use, no effects are

observed during tasks involving the use of laser pointers.• This is why laser pointers are used as control conditions.

• Thus, tool-use based changes may depend on extended reachability, i.e. an expansion of action capacities.

• But extended reachability involves both distal action (manipulation) and distal perception.• E.g. blind man’s cane!

• What remains unclear is which of these two aspects (or both?) causes the qualitative transformations.

Distal perceivability• Is it the fact that elongated tools enable the subject to

perceive at a distance?

• For example, like a blind person using a cane to perceive what is in front of him?

Distal manipulability• Or is it because elongated tools enable us to perform

actions at a distance? • For example, such as when a rake is used to retrieve an

object that is outside of reach?

Distal action and/or perception?• Comparisons between elongated tools and laser pointers

cannot distinguish between distal perceivability and manipulability.

• The problem: Elongated tools enable both of these aspects simultaneously, while laser pointers enable neither of them.

• The question: Which of these two aspects are necessary and/or sufficient to account for the observed effects of elongated tool-use?

Differentiating ‘reachability’

Distal movement and Distal sensation

Distal movement alone

Distal sensation alone

Elongated tool (e.g. cane) ? ?

What kind of tool is the ET?• Distal perception? Yes.

• Subjects trained in using this device readily report a transformation in their perceptual experience that involves the appearance of things ‘out there’ in the world. [elongated tool]

• Distal manipulation? No.• In contrast to using elongated tools, this perception at a distance is

not accompanied by a change in the potential for directly acting at a distance. [laser pointer]

• How does using the ET transform our world and body?

Testing the body image

Random tactile sensation induced by experimenter.

Table top to block sight.

Test: Where do participants perceive the sensations to be in relation to the location on their physical arm?

Participants ‘laser’ point to felt location of stimulus.

Deviation Score Analysis• The difference between actual and perceived length of each

participant's arm after completing the maze task was used to calculate a deviation score. This produced 3 values:• Distance between shoulder and elbow, shoulder and wrist, shoulder

and fingertip of middle finger.

• A between-within MANOVA was used to assess the differences in scores between trained and untrained groups, before and after the task. • Independent variable was group (trained vs. untrained). • Dependent variables were the deviation scores.

• There was no statistically significant difference between trained and untrained on any deviation scores F(1,20) = 0.008; p = 0.932.

Perceived distances of sensations

McGann, Froese, Bigge, Spiers, and Seth (2011)

No unconscious change in arm length

• Thus, we found no evidence in support of a change in participants’ implicit feeling of arm length.

• There are many issues with this result:• Absence of evidence does not equal evidence of absence• Problems with experimental procedure

• Lack of positive control condition.

• Importantly, this is in contrast to other tool-use studies, like the rake-based study of Cardenali, et al. (2009).• But need to replicate their procedures more accurately.

McGann et al. (2011)

No conscious change in arm length, either

Statement Disagree Neither D. or A. Agree

... using the device to found my way through the maze was like 'touching'.

10 4 8

... I experienced the contours of the maze out in the world as if my arm itself had become extended.

9 8 5

It seems that there was generally no clearly defined experience of the touching arm becoming extended.

Given that many participants reported distal perception, this may make the ET experience closer to vision?

Neither touch, nor vision. Then what?Statement Disagree Neither D. or A. Agree

... using the device to find my way through the maze was like 'touching'.

10 4 8

... using the device to find my way through the maze was like 'seeing'.

15 1 6

... using the device to find my way through the maze was like ’feeling'.

5 1 16

The quality of the perceptual experience is ambiguous.Reflecting the ambiguity of the philosophical debate!

This suggests a creative transformation, not simply a substitution between already existing modalities.

Tools transform world and body

• Intentional use of elongated tools changes how we experience our world and our own embodiment.

• Our experiments resulted in refined hypotheses:

• How we experience our world depends on our potential for effective sensation.

• How we experience our body depends on our potential for effective manipulation.• Gallese and Sinigaglia (2010; 2011)

• Both of these potentials of our living bodies can be transformed by intentional tool use.

Further Research• Replicating and verifying the results:

• Testing more participants to increase sample size.• Null result: Repeat task, but with elongated tool as a control.• More training: will there still be no effects after almost 1h?

• More refined questions:• Is there a significant difference between active and passive

changes in subject-environment relations?• E.g. Mossio and Taraborelli (2008)

• Are there ways of objectively measuring when the ET has become experientially transparent for the user?

• Possibility of analyzing sensorimotor time series.• E.g. by looking for 1/f noise, see Dotov et al. (2010)

• Other questions?

Homework• New course unit! Start reading:

• Froese, T., & Gallagher, S. (2012). Getting interaction theory (IT) together: Integrating developmental, phenomenological, enactive, and dynamical approaches to social interaction. Interaction Studies, 13(3), 436-468

• Optional (more detailed account of the mainstream):

• Gallagher, S. (2001). The practice of mind: Theory, simulation or primary interaction? Journal of Consciousness Studies, 8(5-7), 83-108

References• Auvray, M., & Myin, E. (2009). Perception with compensatory devices: From sensory substitution to

sensorimotor extension. Cognitive Science, 33, 1036-1058• Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile

image projection. Nature, 221, 963-964• Block, N. (2003). Tactile sensation via spatial perception. Trends in Cognitive Sciences, 7(7), 285-286. • Dotov, D. G., Nie, L., & Chemero, A. (2010). A demonstration of the transition from ready-to-hand to unready-to-

hand. PLoS ONE, 5(3), e9433. doi: doi:10.1371/journal.pone.0009433• Froese, T., Gould, C., & Barrett, A. (2011). Re-viewing from within: A commentary on first- and second-person

methods in the science of consciousness. Constructivist Foundations, 6(2), 254-269• Froese, T., McGann, M., Bigge, W., Spiers, A., & Seth, A. K. (2012). The Enactive Torch: A new tool for the

science of perception. IEEE Transactions on Haptics, 5(4), 365-375• Froese, T., & Spiers, A. (2007). Toward a phenomenological pragmatics of enactive perception Enactive/07:

Proceedings of the 4th International Conference on Enactive Interfaces (pp. 105-108). Grenoble, France: Association ACROE

• Gallese, V., & Sinigaglia, C. (2010). The bodily self as power for action. Neuropsychologia, 48, 746-755• Gallese, V., & Sinigaglia, C. (2011). How the body in action shapes the self. Journal of Consciousness Studies,

18(7-8), 117-143• McGann, M., Froese, T., Bigge, W., Spiers, A., & Seth, A. K. (2011). The use of a distal-to-tactile sensory

substitution interface does not lead to extension of body image. BIO Web of Conferences, 1(00060). doi: 10.1051/bioconf/20110100060

• Mossio, M., & Taraborelli, D. (2008). Action-dependent perceptual invariants: From ecological to sensorimotor approaches. Consciousness and Cognition, 17(4), 1324-1340

• Noë, A. (2004). Action in Perception. Cambridge, MA: The MIT Press• Prinz, J. (2006). Putting the brakes on enactive perception. Psyche, 12(1), 1-19