59
THE ENCODING OF PARTS AND WHOLES IN THE VISUAL CORTICAL HIERARCHY JOHAN WAGEMANS LABORATORY OF EXPERIMENTAL PSYCHOLOGY UNIVERSITY OF LEUVEN, BELGIUM COLLOQUIUM, LMU MÜNCHEN, JUNE 19, 2013
THE ENCODING OF PARTS AND WHOLES
IN THE VISUAL CORTICAL HIERARCHY
JOHAN WAGEMANS
LABORATORY OF EXPERIMENTAL PSYCHOLOGY UNIVERSITY OF LEUVEN, BELGIUM
COLLOQUIUM, LMU MÜNCHEN,
JUNE 19, 2013
Some examples
Aviezer, Trope & Todorov (2012). Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science, 338(6111), 1225-1229.
Aviezer, Trope & Todorov (2012). Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science, 338(6111), 1225-1229.
Aviezer, Trope & Todorov (2012). Body cues, not facial expressions, discriminate between intense positive and negative
emotions. Science, 338(6111), 1225-1229.
Aviezer, Trope & Todorov (2012). Body cues, not facial expressions, discriminate between intense positive and negative
emotions. Science, 338(6111), 1225-1229.
Visual hierarchy
• features
• parts
• wholes, e.g. – objects – faces – bodies – scenes
Cortical hierarchy: Mainstream view
based on single-unit recordings (Hubel & Wiesel) tuning properties of different types of cells in different
areas of the brain functional specialization hierarchical organization
confirmed in human fMRI (modules, maps) standard view in several approaches
Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.
Grill-Spector, K., & Malach, R. (2004). The human visual cortex. Annual Review of Neuroscience, 27, 649-677.
Serre, T., Oliva, A., & Poggio, T. (2007). A feedforward architecture accounts for rapid categorization. Proceedings of the National Academy of Science of the USA, 104, 6424-6429.
Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.
Grill-Spector, K., & Malach, R. (2004). The human visual cortex. Annual Review of Neuroscience, 27, 649-677.
Serre, T. et al. (2007). A feedforward architecture accounts for rapid categorization. PNAS, 104, 6424-6429.
Alternative views possible
Gestalt theory: Berlin school
Max Wertheimer (1880-1943)
Kurt Koffka (1886-1941)
Wolfgang Köhler (1887-1967)
Alternative views possible
Gestalt theory: Berlin school wholes are more than the sum of the parts wholes are different from the sum of the parts 2-sided dependency wholes come first (e.g., global precedence effect)
Alternative views possible
more recent views Hochstein, S., & Ahissar, M. (2002). View from the top:
Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804.
Bar, M. et al. (2006). Top-down facilitation of visual recognition. Proceedings of the National Academy of Science of the USA, 103, 449-454.
interesting characteristics from viewpoint of Gestalt theory wholes come first highly interactive highly dynamic
Hochstein, S., & Ahissar, M. (2002). View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804.
Bar, M. et al. (2006). Top-down facilitation of visual recognition. PNAS, 103, 449-454.
The problem
• How to understand the relationships between
parts and wholes in visual experience
• How to understand the encoding of parts and wholes in the hierarchy of visual cortex
• How to understand the relationships between cortical encoding and visual experience
My proposal
• There are different types of “Gestalts” with their own
relationships between parts and wholes, both in visual experience and in their neural encoding
• Some Gestalts seem to be encoded in low-level areas based on feedback from higher-order regions – Kourtzi, Z., Tolias, A. S., Altmann, C. F., Augath, M., &
Logothetis, N. K. (2003). Integration of local features into global shapes: Monkey and human fMRI studies. Neuron, 37, 333-346.
– Murray, S. O., Boyaci, H., & Kersten, D. (2006). The representation of perceived angular size in human primary visual cortex. Nature Neuroscience, 9, 429-434.
My proposal
• There are different types of “Gestalts” with their own
relationships between parts and wholes, both in visual experience and in their neural encoding
• Some Gestalts seem to be encoded in lower-level areas based on feedback from higher-level areas
• Other Gestalts seem to be encoded in higher-level areas, while the parts are encoded in lower-level areas – without suppression of the parts – with suppression of the parts
Preservative versus eliminative Gestalts
1. “preservative Gestalts” – functional “wholes” arise spontaneously and
“parts” become less functional – but the encoding of these “wholes” at higher levels
of the cortical hierarchy does not suppress the encoding of the “parts”
2. “eliminative Gestalts” – “wholes” dominate and “parts” disappear from
experience – “wholes” emerge in higher areas of the brain and
encoding of “parts” is then suppressed
Preservative Gestalts
• excellent example: configural-superiority effect
• Pomerantz et al. key papers:
– Pomerantz, J. R., Sager, L. C., & Stoever, R. J. (1977). Perception of wholes and their component parts: Some configural superiority effects. Journal of Experimental Psychology: Human Perception and Performance, 3, 422-435.
– Pomerantz, J. R., & Portillo, M. C. (2011). Grouping and emergent features in vision: Toward a theory of basic Gestalts. Journal of Experimental Psychology: Human Perception and Performance, 37, 1331-1349.
• neural basis?
Kubilius et al. (2011)
• Kubilius, J., Wagemans, J., & Op de Beeck, H. P. (2011).
Emergence of perceptual Gestalts in the human visual cortex: The case of the configural superiority effect. Psychological Science, 22(10), 1296-1303.
• behavioral results
• fMRI decoding results
Behavioral results
parts corner whole
Scanning protocol
fMRI results: Retinotopic mapping
MVPA: decoding
fMRI results: decoding
fMRI results: decoding
Discussion
• behavioral configural-superiority effect
• neural configural-superiority effect: – better coding of “wholes” than “parts” in higher shape-
selective regions – better coding of “parts” than “wholes” in lower-level
retinotopic regions
• general conclusions: – at least some Gestalts emerge only at higher stages of
visual information processing – feedforward processing may be sufficient to produce
some Gestalts
Two examples of eliminative Gestalts
• Motion silencing – Suchow, J. W., & Alvarez, G. A. (2011). Motion silences
awareness of visual change. Current Biology, 21(2), 140-143. doi:10.1016/j.cub.2010.12.019
– Poljac*, E., de-Wit*, L., & Wagemans, J. (2012). Perceptual wholes can reduce the conscious accessibility of their parts. Cognition, 123, 308-312. (*joint first authors) doi:10.1016/j.cognition.2012.01.001
• Bistable diamond – Fang, F., Kersten, D., & Murray, S. O. (2008). Perceptual
grouping and inverse fMRI activity patterns in human visual cortex. Journal of Vision, 8(7):2, 2-9. doi: 10.1167/8.7.2
– de-Wit, L. H., Kubilius, J., Wagemans, J., & Op de Beeck, H. P. (2012). Bi-stable Gestalts reduce activity in the whole of V1 not just the retinotopically predicted parts. Journal of Vision, 12(11):12, 1-14. doi:10.1167/12.11.12
Suchow & Alvarez (2011)
• Suchow, J. W., & Alvarez, G. A. (2011). Motion silences awareness of visual change. Current Biology, 21(2), 140-143. doi:10.1016/j.cub.2010.12.019
• “Best Illusion of the Year 2011”
Demonstration
Demonstration
More demonstrations
Methods
• Stimuli: – 100 dots – first stationary, then rotating back and forth for 30° – 2 phases alternating every 3 s
• Task: – observers had to adjust the rate of change during the
stationary phase to match the apparent rate of change in the moving phase
– rate of change (“silencing factor”) between 0.1 (static perceived as changing slower) and 10 (static perceived as changing faster)
Results
Interpretation
• Suchow & Alvarez:
– local mechanisms with small receptive fields – because a fast-moving object spends little time at any
one location, a local detector is afforded only a brief window in which to assess the changing object
• alternative interpretation: – objecthood – when a good “whole” is formed, the details of the
“parts” are fundamentally less accessible to conscious perception
Our study
• Poljac*, E., de-Wit*, L., & Wagemans, J. (2012). Perceptual wholes can reduce the conscious accessibility of their parts. Cognition, 123, 308-312.
(*joint first authors) doi:10.1016/j.cognition.2012.01.001
• motivation: to test this alternative interpretation with biological motion
Why biological motion?
• a prototypical case of a complex hierarchical stimulus (Johansson, 1973; Cutting & Proffitt, 1982) – multiple elements, each with their own spatio-temporal
trajectories – organized quickly and efficiently in a hierarchical configuration,
in which the motion of the local elements are coded relative to a more global structural description
• the perceptual Gestalt is constructed automatically by the
visual system (Thornton & Vuong, 2004)
• the construction of the perceptual whole implies a more efficient representation of the relationships between the parts (Tadin et al., 2002)
• inversion allows control over low-level motion trajectories (Sumi, 1984; Pavlova & Sokolov, 2000)
Demonstrations
Demonstrations
Demonstrations
Methods
• Stimuli:
– motion captured point-light treadmill walkers (Vanrie & Verfaillie, 2004)
– 70 colored dots (“confetti walker”) – upright, inverted, phase-scrambled
• Task: adjust rate of change in the test figure until it matches the rate of change in the comparison figure
• Direct comparison of dynamic and static – comparison figure: scrambled – test figures: upright or inverted
Results
Discussion
• on top of the effect of static vs moving, there is a clear effect of configurality (“goodness” of the whole percept)
• cost of objecthood: the more strongly the parts are integrated into the perception of a whole object, the less accessible the changing features of the parts are (e.g., also embedded figures)
Eliminative Gestalts
• excellent example: bistable diamond
• Murray et al. key papers:
– Murray, S. O., Kersten, D., Olshausen, B. A., Schrater, P., & Woods, D. L. (2002). Shape perception reduces activity in human primary visual cortex. Proceedings of the National Academy of Sciences, 99, 15164-15169.
– Fang, F., Kersten, D., & Murray, S. O. (2008). Perceptual grouping and inverse fMRI activity patterns in human visual cortex. Journal of Vision, 8(7):2, 2-9. doi: 10.1167/8.7.2
Demonstration
Nice features
• perceptual bi-stability: – “parts” seen to move vertically – “whole” seen to move horizontally
• switching relatively slow, perceptual states rather clear
• stable individual differences
• studied rather extensively at psychophysical level, e.g. – Lorenceau, J. & Shiffrar, M. (1992). The influence of
terminators on motion integration across space. Vision Research, 32, 263-273.
Murray et al.: Design
• present bistable diamonds
• ask observers to indicate perception of “parts”
(line segments) or “whole” (diamond)
• record BOLD responses (fMRI) in different areas and relate these to the reported percepts
Murray et al.: Results
Murray et al.: Results
Discussion
• convincing demonstration of inverse activity patterns in V1 and LOC
• interpretation? – perception of “parts” suppressed by perception of
“whole” – predictive coding framework: “explaining away”
Discussion
• however – in a recent follow-up study, we have shown that the
reduction of activity in V1 is global, not retinotopically specific
• de-Wit, L. H., Kubilius, J., Wagemans, J., & Op de
Beeck, H. P. (2012). Bi-stable Gestalts reduce activity in the whole of V1 not just the retinotopically predicted parts. Journal of Vision, 12(11):12, 1-14. doi:10.1167/12.11.12
Conclusion
• the encoding of parts, wholes and their relationships constitutes a serious challenge to the visual system
• the visual system appears to have developed flexible mechanisms with different characteristics – sometimes wholes are encoded in low-level areas
(feedback?) – sometimes wholes are encoded in high-level areas, while
parts are preserved in low-level areas – sometimes wholes are encoded in high-level areas, while
parts are suppressed in low-level areas
• further research is needed to establish the specific properties of these cases (computational reasons, boundary conditions, etc.)
THANK YOU
WWW.GESTALTREVISION.BE
The encoding of parts and wholes �in the visual cortical hierarchy�Some examplesSlide Number 3Slide Number 4Slide Number 5Slide Number 6Slide Number 7Slide Number 8Slide Number 9Slide Number 10Visual hierarchyCortical hierarchy: Mainstream viewFelleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.Grill-Spector, K., & Malach, R. (2004). The human visual cortex. �Annual Review of Neuroscience, 27, 649-677.Serre, T. et al. (2007). A feedforward architecture accounts for rapid categorization. PNAS, 104, 6424-6429.Alternative views possibleAlternative views possibleAlternative views possibleHochstein, S., & Ahissar, M. (2002). View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804.Bar, M. et al. (2006). Top-down facilitation of visual recognition. �PNAS, 103, 449-454.The problemMy proposalMy proposalPreservative versus eliminative GestaltsPreservative GestaltsKubilius et al. (2011)Behavioral resultsScanning protocolfMRI results: Retinotopic mappingMVPA: decodingfMRI results: decodingfMRI results: decodingDiscussionTwo examples of eliminative GestaltsSuchow & Alvarez (2011)DemonstrationDemonstrationMore demonstrationsMethodsResultsInterpretationOur studyWhy biological motion?DemonstrationsDemonstrationsDemonstrationsMethodsResultsDiscussionEliminative GestaltsDemonstrationNice featuresMurray et al.: DesignMurray et al.: ResultsMurray et al.: ResultsDiscussionDiscussionConclusionThank You
