Reconstruction of Subjective Surfaces from Occlusion Cues Naoki Kogo 1 , Christoph Strecha 1 , Rik Fransen 1 , Geert Caenen 1 , Johan Wagemans 2 , and Luc Van Gool 1 1 Katholieke Universiteit Leuven, ESAT/PSI, B-3001 Leuven, Belgium [email protected]http://www.esat.kuleuven.ac.be/psi/visics 2 Katholieke Universiteit Leuven, Department of Psychology B-3000 Leuven, Belgium Abstract. In the Kanizsa figure, an illusory central area and its con- tours are perceived. Replacing the pacman inducers with other shapes can significantly influence this effect. Psychophysical studies indicate that the determination of depth is a task that our visual system con- stantly conducts. We hypothesized that the illusion is due to the mod- ification of the image according to the higher level depth interpreta- tion. This idea was implemented in a feedback model based on a surface completion scheme. The relative depths, with their signs reflecting the polarity of the image, were determined from junctions by convolution of Gaussian derivative based filters, while a diffusion equation recon- structed the surfaces. The feedback loop was established by converting this depth map to modify the lightness of the image. This model created a central surface and extended the contours from the inducers. Results on a variety of figures were consistent with psychophysical experiments.
Transcript
Reconstruction of Subjective Surfaces from
Occlusion Cues
Naoki Kogo1, Christoph Strecha1, Rik Fransen1, Geert Caenen1, JohanWagemans2, and Luc Van Gool1
http://www.esat.kuleuven.ac.be/psi/visics2 Katholieke Universiteit Leuven, Department of Psychology
B-3000 Leuven, Belgium
Abstract. In the Kanizsa figure, an illusory central area and its con-tours are perceived. Replacing the pacman inducers with other shapescan significantly influence this effect. Psychophysical studies indicatethat the determination of depth is a task that our visual system con-stantly conducts. We hypothesized that the illusion is due to the mod-ification of the image according to the higher level depth interpreta-tion. This idea was implemented in a feedback model based on a surfacecompletion scheme. The relative depths, with their signs reflecting thepolarity of the image, were determined from junctions by convolutionof Gaussian derivative based filters, while a diffusion equation recon-structed the surfaces. The feedback loop was established by convertingthis depth map to modify the lightness of the image. This model createda central surface and extended the contours from the inducers. Resultson a variety of figures were consistent with psychophysical experiments.
Reconstruction of Subjective Surfaces from
Occlusion Cues
No Author Given
No Institute Given
Abstract. In the Kanizsa figure, an illusory central area and its con-tours are perceived. Replacing the pacman inducers with other shapescan significantly influence this effect. Psychophysical studies indicatethat the determination of depth is a task that our visual system con-stantly conducts. We hypothesized that the illusion is due to the mod-ification of the image according to the higher level depth interpreta-tion. This idea was implemented in a feedback model based on a surfacecompletion scheme. The relative depths, with their signs reflecting thepolarity of the image, were determined from junctions by convolutionof Gaussian derivative based filters, while a diffusion equation recon-structed the surfaces. The feedback loop was established by convertingthis depth map to modify the lightness of the image. This model createda central surface and extended the contours from the inducers. Resultson a variety of figures were consistent with psychophysical experiments.
1 INTRODUCTION
A well-known figure that provokes an illusory perception, known as the Kanizsafigure, has been a key instrument to investigate the perceptual organization ofour brain (see [1] [2] [3] for review). In this paper, a model with biologicallyplausible architecture was built to reproduce these subjective properties. In thefollowing subsections, some principles implemented in this model are explained.
Fig. 1. Variations of Kanizsa image (A). The illusion disappears in B and F, and isweaker in C. The central square appears lighter in A and I, and darker in H and J.
1.1 Depth recognition task
Fig.1 shows the Kanizsa square (1A) as well as its variations with modifiedinducers. Replacing the ‘pacman’ inducers by crosses as in 1B causes the dis-appearance of the illusory perceptions. Fig.1C rather shows a weakening of theeffect. The key question is what brain function in our visual system causes thiscontext sensitive perception. We hypothesized that the constantly conductedtask in our visual system, the determination of the depth order of objects, isresponsible for this phenomenon. The key property of the Kanizsa image is thatit is constructed such that the depth interpretation at the higher level vision isin conflict with the physical data of the input. A visual agnosia patient could notperceive this illusion unless it was presented with stereo disparity [4], presumablydue to the lack of brain functions to detect non-stereoscopic pictorial depth cues.Also, when a Kanizsa image is created as an isoluminant figure, the illusion is notevoked [5]. De Weert [6] showed that our depth recognition mechanism is colourblind and its function is suppressed severely with isoluminant images. Finally,Mendola et.al. [7] reported on the active area of the brain when human subjectsare seeing Kanizsa images. The results indicated that the area corresponded tothe active area during depth recognition tasks. These results suggest that thedepth recognition is playing the fundamental role in the Kanizsa illusion.
1.2 Usage of differentiated form of signals
From the beginning of history of neural recording from the visual cortex, it hasbeen known that the neurones respond to borders but not to the interior betweenthem [8]. This neural behavior has been considered as somewhat puzzling sinceit is a fact that we perceive the interior region. One plausible explanation is thatthe neurones encode the differentiated signal and in this way the original imagecan always be reconstructed by integration. In other words, the interior infor-mation is indeed preserved in the differentiated form of the signal. To conductdepth recognition in a computer vision model, this provides a quite convenientapproach. The model only needs to focus on the local properties of the image todetermine the relative depth between the immediately neighbouring loci. Aftercollecting the individual local information, the macroscopic features of the imagecan be reconstructed by integrating the differentiated signal.
1.3 Feedback system
From perceptual experiments, it is clear that higher level visual processing isinvolved in the illusion. Indeed, lesions (by artificial or accidental brain dam-age) in the higher level visual cortex resulted in the elimination of such illusions[4][9]. However, neural activities at the lower level already show responses toillusory contours [10]. This apparent paradox can be solved if a feedback con-nection from higher level to lower level is considered. Psychophysical studieshave demonstrated feedback effects on lower level vision by biasing its responseproperties [11] [12]. Electrophysiologically, it has been shown that the feedbackconnections from the higher level visual cortex to V1 and V2 can modify thereceptive field properties of the neurones [13] [14]. If indeed the feedback pro-jection changes the lower level responses according to macroscopically detectedfeatures, it would create a mechanism of “biased” perception through the feed-back loop. We assume that this biased control in the feedback system must bequite effective when the image is ambiguous, as is the case in illusory figures.
1.4 Depth lightness conversion
It is important to note that the illusion is a result of a modification of the physi-cally defined input image measured in intensity of lightness. If depth perceptionplays a key role in the illusion, as we hypothesise, then, the modification of theimage has to reflect the perceived depth. This is justified by the fact that thereis a dependency of the perception of lightness on the perception of depth [15].Once this conversion from depth to lightness is established, the image can bemodified accordingly. Apparently, this is a necessary step for the feedback sys-tem described above to iteratively modify the image’s lightness measurement.However, this modification has to be done with caution. As shown in Fig.1H,and more prominently in 1J, when the figure has white inducers on black orgray background, the central area is perceived darker, i.e. opposite to the effectin Fig1.A and I. The decrementing configuration of the figure (black on white)
creates “whiter than white” effect, while the incrementing one creates “blackerthan black” effect in the central area. The modification of the image has to reflectnot only the depth but also the polarity of the image.
1.5 Model architecture and comparison with earlier models
In summary, we developed a model that integrated the hypotheses discussedabove: (1) the depth recognition task is involved in the creation of the illusion,(2) analysis of local properties gives relative depth information that is repre-sented by the differentiated form of the signal, (3) global integration of thesignal determines the perceived depth which, in turn, modifies the image by atop-down projection in the feedback loop, (4) this feedback modification is con-ducted through a mechanism that links between lightness perception and depthperception.Several earlier models for the Kanizsa phenomenon have focused on edge com-pletion driven by local cues (see [16], [17] for example). As a result, some modelscould not distinguish between modal subjective contours (the contours along thecentral square) and amodal contours (the missing part of the circular contours).It is also very likely that they can not differentiate between the cases in Figs 1Aand 1B. Models that were constructed by a surface reconstruction scheme basedon the judgment of occlusions, on the other hand, showed more robust propertiesreflecting the differences between the variations of the figure [18] [19]. Our model,which is based on the analysis of junction properties to construct a depth map,will be closer to the surface completion models. We, however, corroborate therather intuitive point of departure of surface completion models by argumentsobtained from perception research, as mentioned above. In addition, our model,differing from other surface completion models, is based on filter convolutionsthat are constructed from derivatives of Gaussian functions, which are known tobe relevant to the biological system.
The rest of the paper is organized as follows: Section 2 describes the model inmore detail, section 3 shows some results obtained with our model, and section 4concludes the paper with a discussion.
2 THE MODEL
2.1 Junction properties
In our model, it is assumed that the only depth cue in the simplified imagelike the Kanizsa figure is provided by the property of junctions (or concavityin general) indicating the occurrence of occlusions. The model has to determinethe depth relationship between areas s1 and s2 in Figure 2, for instance. Inthe model, it is assumed that the area (s1) on the side of the narrower angle(less than 180◦) of the junction (j1) is the “occluding” area and the other onewith the wider angle (more than 180◦) being the “occluded” area (s2). In otherwords, such junction is taken as a cue that s1 is closer to the viewer than s2.
There are two other junctions in the inducer (j2 and j3), which act as oppositecues. However, they contain curved borderlines in the Kanizsa figure, which willweaken the strengths of these cues. This is in contrast with cases B and C inFigure 1, where the junctions all consist of long straight lines. This difference isreflected in the amplitude of the junction signal simply because the elongatedfilters used in the model will give smaller responses to curved lines.
s1
s2 j1
+-j2
j3
s1
s2 j1
j2
j3
Fig. 2. “occluding” (+) and “occluded” (-) areas near junctions of pacman inducer(left) in the Kanizsa figure and non-determined areas in L-shape corner (right) of thefour corners figure (Fig.1C).
2.2 Filter convolutions: Border map and junction detection
In the first stage of the algorithm, we (1) detect the border of the image by afirst convolution with elongated 2-D Gaussian derivative (GD) filters (section2.2), (2) compute the location of junctions and create the “differentiated signal”to indicate the relative depth by a second convolution applied to the detectedborderlines (section 2.3). Next, we integrate the “relative depth” informationto create a global depth map using an anisotropic diffusion equation (section2.4). Finally, we modify the input to create a feedback system by linking depthinformation to lightness perception (section 2.4).To detect the borders, the image is convolved with GDs of different orientations.The result is called the “signed first convolution”, Fs. The border map, Fb,is defined as the absolute value taken from Fs (eq.1 and 2). For the junctiondetection, a “half” Gaussian derivative (hGD) is created to generate a sharpsignal at a junction of known orientation so that it aids the creation of therelative depth map described below. This filter was created as follows. A firstGD filter was elongated perpendicular to the direction of the differentiation. Asecond GD filter whose differentiation and elongation directions are the same asthe elongation direction of the first GD filter, was created and rectified. The firstGD filter was multiplied by this second “positive only” GD filter. Depending onthe polarity of the first GD filter, “right-handed” and “left-handed” filters werecreated (Fig.3). Two filters with opposite polarities and 90◦ angle differences(angle for left-handed filter being 90◦ incremented from the angle for right-handed filter1) are always used as a pair. After the convolutions with the borders
1 The “angle” of filter indicates the direction of the filter from the origin toward theelongation. When its positive portion is on the right side of the direction, it is called
detected above (Fb), only the positive portions of the results are taken. Theresults from the two filters are multiplied to signal the location of the junction.The amplitude of this response reflects the straightness of the two borderlinesthat belong to the junction as well as the contrast between two adjacent areas.
Next, the relative depth map (RDM) is constructed using the values alwayson the side of the narrower angle of the junction, with their signs reflecting thepolarity of the image (eq.1). First, a convolution (∗) of Gaussian (G) to the junc-tion map (J) is made to determine the territory of the junction. Here, territorymeans the area where the local analysis of the junction property can influencethe result. The square of this value is multiplied with the “second convolution ofborder”, Sb, obtained by convolving Fb with hGD filters. By rectifying the result(rect), it creates the relative depth value (Rd) along each border. However, toreflect the polarity of the image (i.e. incrementing and decrementing configura-tion of the image, see section 1.4), it has to create the polarized relative depthvalue (Rp), instead. This is achieved by multiplying Rd with the “signed sec-ond convolution”, Ss, obtained by convolution of Fs with a hGD (eq.2). Addingthis value from all angle combinations creates the polarized relative depth map(pRDM).
Rd = rect((G ∗ J)2 × Sb) Sb = hGD ∗ Fb (1)
Rp = Rd × Ss Ss = hGD ∗ Fs (2)
2.4 Integration: surface reconstruction and the feedback loop
In our model, the integration of the pRDM signal is done by using a modifiedanisotropic diffusion equation developed in our group [20] as shown in eq.3. With
right-handed and if it is on the left, left-handed. The angle is measured counter-clockwise from x direction. The angles of the filters used in this model are from 0◦
to 270◦ in 90◦ increments.
this method, the positive and negative values in the pRDM spread in 2-D spacewhere they are restricted by the borders of the original image, Fb.
∂f
∂t= div(Cf∇f) − λ(1 − Fb)(f − pRDM) Cf = e
−
FbKf
2
(3)
Here, f is the result of diffusion and λ and Kf are constants. This diffusioncreates the depth map that also reflects polarity of the figure (polarized depthmap, PDM) as shown in Fig.4B. As discussed in section 1.4, the perceived depthcan be used to modify the image to reproduce the perception of the figure. Thiswas done by first obtaining the product of the PDM with a conversion factorfrom depth to lightness, α (eq.4), and then giving it an offset. Since the PDMhas no values at the background area which corresponds to the fact that nosubjective modifications of the image take place in this area, this offset valuewas determined to be 1 so that the value in the background area indicates nochange, as shown in Fig.4C. The so-called modification factor and is multipliedto the original image (I0) to create the modified image (I, Fig.4D). This modifiedimage is then used to feed the next iteration of the feedback loop.
I = I0 × (1 + αf) (4)
0
255
1
0
0
255
1
0
0 0
0
255
0
255
A
C
D
B
0
255
1
0
0
255
1
0
0 0
0
255
0
255
A
C
D
B' 0 0
Fig. 4. Polarized depth map (PDM) and the computation of the modification factorexplained in a 1-D plot. A: Original image plotted in gray scale. The left image indicatesthe decrementing configuration (white background and two black objects) while theright image indicates the opposite polarity of the figure. B: PDM, C: Modificationfactor, D: Modified image, B’: Non-polarized depth map. Note that the change in thecentral ares is stronger in decrementing configuration as indicated by arrows.
3 RESULTS
The border and junction maps of the Kanizsa and the four corners figures (Fig.5Aand C) are shown in Fig.5. Only the junctions of the same direction (pointing
Fig. 5. Border and junction maps created by the two filters of 0 & 90◦. The positionof the detected junctions are shown in the border map (X). Left Kanizsa figure andright the four corners figure.
to 45◦) detected by the right-handed 0◦ and left-haded 90◦ filter pair are shown.By repeating the procedure for different angle combinations, all the junctionspresent in these figures are detected correctly. Adding all together yields thepRDM (Fig.6, top). In the Kanizsa figure, the signals near the middle junctionsof the inducers are stronger than the ones in the four corners figure due to the factthat the competing information from middle and side junction cancel out eachother in the four corners figure while in the Kanizsa figure, the information fromthe middle junction wins. The borderline-restricted diffusion of pRDM createdthe surface with different height (PDM, Fig.7, bottom). The central square inKanizsa was either lifted (D) or lowered (E) from the ground depending on thepolarity of the image (note the reversal of signs in corresponding pRDM in Fig.7Aand B). The modification factor was created from the PDM and multiplied tothe input image, resulting in the modified output image (Fig.7 B, plotted as“reflectance map”). Fig.8 shows the responses to variations of the Kanizsa figure(Fig.1). Importantly, (1) the four crosses figure did not create contrast betweenthe central area and the background, (2) the four corners figure created a lightercentral area in lesser extent than the Kanizsa figure, (3) change in the skeltonizedKanizsa was negligible, (4) the response reflected the polarity of the image withlesser extent in the incrementing configuration. The result is fed into the first stepof the border detection, and the whole procedure is repeated. The result is shownin Fig.9. This clearly shows that the central square became more prominent afterthe iteration of the feedback loop. Through the iterations of the feedback loop,the extension of the edges from the inducers is observed (Fig.10) due to thefact that the contrast now exists between the central square and the backgroundgenerating responses in the convolution with GD filters.
4 DISCUSSION
Our model was designed to detect the relative depth of surfaces based on junc-tion properties. It is somewhat related to earlier surface completion models [18].Our model, however, is based on filter convolutions and therefore responds tothe contrast of the image quantitatively. In addition, the model induces lightnesseffects (perceived reflectance) based on the measured depth. It also reflects the
A B C
D E F
Fig. 6. A, B and C: Polarized relative depth map (pRDM) of Kanizsa with decrement-ing (A), and incrementing (B) configuration, and the four corners figure (C). D,E andF: The result of the integration (PDM) of pRDMs from above. Whiter color indicatespositive, and blacker, negative value.
80100
120140
160180
80
100
120
140
160
180
0
50
100
150
200
250
80100
120140
160180
80
100
120
140
160
180
0
50
100
150
200
250
Fig. 7. The input (A) and the modified (B) image plotted as a lightness map.
A B C D
E F G
H I
Fig. 8. Output of the model to variations of Kanizsa figures corresponding to thefigures shown in Fig.1, except H and I are responses to incrementing and decrementingKanizsa (side view).
0 1 3 5 7 10
Fig. 9. Development of the central surface of Kanizsa figure through the feedback loop.From left, the original image and the results after iterations.
Fig. 10. Magnified of the border map (side view). In the gap between the pacmaninducers, edge extensions are observed during the iteration of the feedback loop. Theright most figure is a magnified top view from the 10th iteration.
polarity of the figure, depending on whether black inducers are on a white back-ground or vice versa. The model showed quite robust responses which stronglycorrelates to psychophysical data from the different types of Kanizsa figures. Themodel, for instance, enhanced the lightness in the central area in decrementingconfiguration (whiter than white effect) or reduced it (blacker than black effect)in incrementing configuration (see Fig.7C and D). It also correctly respondedto the skeltonized Kanizsa image (Fig.1) by not producing the subjective mod-ification due to the filter convolution only giving a small signal for this imageconstructed with thin lines. Finally, through the iteration of the feedback loop,the edges of the pacman inducers started to extend which mimics the contourcompletion effect surrounding the central square.To improve the robustness of our model, some additional studies are necessary.First, the enhanced responses to the end-stopped portion of the lines needs tobe introduced by adding a surrounding “inhibitory field” (negative area) to thehGD filter. The responses of the model with this “end-stopped filter” to Kanizsatype images is being investigated along with its responses to T junctions. In thispaper, only the junctions with 90◦ angle gap are detected. The new algorithmto detect junctions with various angle gaps and orientations is being developed.The quantitative aspects of the responses are being analysed in conjunction withsome psychophysical experiments to achieve the plausible measure of α value andthe gray scaling of the perceived lightness of the image. And finally, the feedbacksystem which dynamically modifies the parameters of the convolution filters isbeing developed to further enhance the effects such as the extension of the edgesfrom the inducers. In addition, the performances of this algorithm with real-lifeimages and more complex illusory figures are being investigated.
tation of Illusory and Real Contours in Human Cortical Visual Areas Revealed byFunctional Magnetic Resonance Imaging. J. Nuerosci. 19 (1999) 8560-8572
[8] Hubel, D.H., Wiesel, T.N.: Ferrier lecture. Functional Architecture of MacaqueMonkey Visual Cortex. Proc. R. Soc. Lond. B Biol. Sci. 198 (1977) 1-59
[9] Huxlin, K.R., Saunders, R.C., Marchionini, D., Pham, H., Merigan, W.H.: Percep-tual Deficits after Lesions of Inferotemporal Cortex in Macaques. Cerebral Cortex10 (2000) 671-683
[10] Peterhans, E, von der Heydt, R.: Mechanisms of Contour Perception in MonkeyVisual Cortex. II. J. Neurosci. 9 (1989) 1749-1763
[11] Mecaluso, E., Frith, C.D., Driver, J.: Modulation of Human Visual Cortex byCrossmodal Spatial Attention. Science 289 (2000) 1206-1208
[12] Aleman, A., Rutten, G.M., Sitskoorn, M.M., Dautzenberg, G., Ramsey, N.F.: Ac-tivation of Striate Cortex in the Absence of Visual Stimulation:an FMRI Study ofSynthesia Neuroreport 12 (2001) 2827-2830
[13] Hupe, J.M., James, A.C., Payne, B.R., Lomber, S.G., Girard, P., Bullier, J.: Cor-tical Feedback Improves Discrimination Between Figure and Background by V1,V2 and V3 Neurons. Nature 394 (1998) 784-787
[14] Wang, C., Waleszczyk, W.J., Burke, W., Dreher, B.: Modulatory Influence ofFeedback Projections from Area 21a on Neuronal Activities in Striate Cortex ofthe Cat. Cerebral Cortex 10 (2000) 1217-1232
