1. INTRODUCTIONEvolution can be seen as a process that stores and accu-mulates knowledge about the physical structure inherentin the environment genetically. These genes then gener-ate by means of self-organizing chemical processes a phe-notype with specific traits. Such traits can be of a morestructural sort (e.g., design and shape of limbs, organs,etc.) or might correspond to typical behavioral or motorpatterns. However, these traits must have increased thefitness of the predecessors of an organism under consid-eration in order to be passed to the offspring. If we focusnow on mammalian visual systems, we can see that dif-ferent animals share common characteristics. For in-stance, the structure of classical receptive fields of corticalsimple cells can be described by Gabor functions.1 An-other example is the oblique effect. This refers to en-hanced perception (in the foveal retina) along the horizon-tal and vertical retinal axes of vision compared with thatof contours at oblique angles. Hereafter we refer to theseas cardinal axes. The effect reveals itself, for instance, inan orientation-dependent contrast sensitivity function orin an increased acuity to discriminate differences in ori-entation if they are close to the cardinal axes.2 Again,the oblique effect can be observed in many different spe-cies, e.g., rats, ferrets, cats, monkeys, and humans,3 al-though those animals see the world from different view-ing angles. This implies that the physical structure ofthe environment provides constraints for the evolutionaryprocess, and it is this structure that also exerts strong in-fluences on the postnatal development of an organism.We now can ask if certain structures of the brain corre-spond to specific boundary conditions for the evolutionaryprocess, and if this turns out to be true, then we can ex-
pect to find them by analyzing the physical structure ofthe environment. An interesting experiment that pro-vides strong support for proceeding in this way was doneby Olshausen and Field.4 They obtained localized, ori-ented, and bandpass receptive fields that resemble theclassical receptive fields of simple cells in the primary vi-sual cortex by training an unsupervised learning algo-rithm with natural images. The network was designedto optimize two constraints. The first one was that therebe an average sparse activation of the network units, andthe second one was that the input be recoverable from theoutput. This amounts to maximization of sparseness andpreservation of information, respectively.
Apart from the evolutionary process, which is analo-gous to learning on a slow (phylogenetic) time scale, thereexists a more flexible learning scheme on an ontogenetic(faster) time scale. For example, if kittens are reared incylinders painted with stripes of only a single orientation,simple cells will adapt by shifting their preferred orienta-tion to the experienced one.5 Thus another importantquestion refers to the time scale of development, i.e.,whether we are faced with a phylogenetic or an ontoge-netic acquired trait. Consider again the rearing of kit-tens in a striped cylinder. If we assume that the pre-ferred orientation of simple cells redistributes accordingto the experienced orientations during growing up, thenbehavior might influence the outcome. Thus head rota-tion may broaden the distribution of cells away from theexperienced orientation (but this seems not to be the case;see Ref. 5).
Another example in which one can assume a link be-tween the physical structure of the environment and astructural trait of the mammalian visual system is the ob-
2000 Optical Society of America
698 J. Opt. Soc. Am. A/Vol. 17, No. 4 /April 2000 M. S. Keil and G. Cristobal
lique effect. Let us first look at the brain. By consider-ing the brain of ferrets, Chapman and Bonheoffer6 foundthat, in accordance with the study of Coppola et al.,7 in allferrets more cortical area is devoted to the processing ofthe cardinal axes of the corresponding foveal region. Thesize of this asymmetry showed some variation across theanimals used in their experiments. Further, Coppolaet al.7 reported that, on the average, approximately 2%more area of the primary visual cortex preferred horizon-tal to vertical orientations. Hereafter we will use theterm anisotropy to denote this property. Our results forthe natural image set are consistent with this neuro-physiological finding for higher spatial frequencies.
Schall et al.8 compared the orientation of dendriticfields of retinal ganglion cells with their correspondingtargets in cats. Again, there was found an overrepresen-tation of the cardinal orientations in both retina and vi-sual cortex. Note that although the visual systems ofcats and ferrets are similar in their structure, they show adifferent developmental state at the time of birth. Thecat visual system is approximately two or three weeksahead in maturity.9,10
Now we can look for corresponding statistical regulari-ties in the physical structure of the environment. Firstindications were provided by Switkes et al.11 They calcu-lated the power spectra of natural images by using opticalFourier analysis. Three classes were established, accord-ing to the landscapes or the environments in which theirimages were taken. These comprised indoor carpenteredscenes (19 pictures), outdoor carpentered scenes (23), andpastoral scenes (15). Under the assumption that visualexperience alone can account for the oblique effect, theyconcluded that ‘‘since the implicit assumption that car-pentered environments present more high frequency hori-zontal and vertical components than do non-carpenteredenvironments is not supported by the data reported here,we may conclude either that environment has no effect inproducing anisotropy, or that if it does, it is by a mecha-nism which is not specific for spatial frequency’’ (Ref. 11,p. 1397). A later study, nevertheless, came up with dif-ferent results.12 In this study Coppola et al. gatheredphotographs by means of adjusting their camera with acarpenter’s level. This was done to achieve correct align-ment with the horizon line. Again, images were catego-rized in sets for indoor, outdoor, and natural scenes.From each class 50 images were selected randomly forfurther analysis. The authors did not mention whetherthey calibrated their images in order to compensate pos-sible artifacts produced by the optical system of the cam-era. Orientation was determined by using 9 3 9 Sobelfilters. This corresponds to the extraction of high-spatial-frequency content. A decreasing ratio of Sobelmagnitude at filter orientations of both 0° and 180° wasobserved along the picture categories indoor, outdoor, andentirely natural. Interestingly, their results seem to beslightly different for 0° and 180° as well as for 90° and270°. The conclusions that Coppola et al. drew give sup-port to the hypothesis that the oblique effect has indeedits origin in the visual environment.
However, two issues are not yet clarified. In additionto their selectivity for a certain orientation, simple cellsalso show a selectivity for spatial frequency. So it is sur-
prising that up to now no one has attempted to clarify therole of spatial frequency in the expression of the obliqueeffect, either neurophysiologically or statistically. Butthis requires a quantification of the oblique effect, and sowe defined two measures, namely, cardinal versus obliqueenergy ratio (COR) and aligned energy ratio (AER). Forinstance, Coppola et al.12 discussed only the high-frequency content of their images (they only briefly men-tioned that their results also seem to hold for down-sampled versions of their images). Furthermore, theyused aligned images, although we cannot tacitly assumethat a developing organism encounters a set of imagesstabilized in this way. In this sense the images that weused were rather unbiased. It will be interesting to seewhether, on the average, alignment is generated ratherthan introduced a priori. One further example is thestudy from van der Schaaf and van Hateren.13 They dis-cussed (although for a different image set from the onethat we were using here) the dependence of energy on ori-entation by using a power spectrum averaged over the im-age set and over spatial frequencies. No distinctionswith respect to image classes were made. Another issuethat we will discuss here is whether the effect was pro-duced during phylogeny or is established ontogenetically.
Methodical differences of our study from the describedprevious studies include the definition of five imageclasses. This allows for a more refined examination of vi-sual environments. Furthermore, by defining two mea-sures based on the oriented energy (OE), we are able toquantify the distribution of energy at cardinal versus ob-lique angles as well as the anisotropy itself of the energyat the cardinal orientations. These measures are theCOR and the AER.
To examine the frequency dependencies of the CORand the AER, we used localized bandpass integration ar-eas onto the two-dimensional power spectrum [or energyspectrum (ES)]. Inverse Fourier transformation of theseareas would yield Gabor-like filter pairs in quadraturephase in the spatial domain. It should be noted that thiswas not supposed to be an attempt to mimic the responseproperties of cortical simple cells. Rather, we chosethese methods to meet the well-known property of self-similarity of natural images14 and to obtain a good reso-lution in both angular and spatial-frequency dimensions.Stated differently, our concern here is to analyze exclu-sively the physical structure of the environment ratherthan to look at the world through the glasses of the mam-malian visual system. We try to find the boundary con-ditions or constraints that were exerted on the evolution-ary process in order to understand its solutions.
Finally, we experimented with three different versionsof the ES for calculating the OE to demonstrate that thespecific functional form that is used to read out the physi-cal information is of no major importance for our results.We used, besides the unmodified (or direct) power spec-trum, a logarithmic and a k2-windowed version (where kis the spatial frequency).
2. IMAGE PROPERTIES ANDCLASSIFICATIONImages were taken from the natural stimuli collection ofthe laboratory of Hans van Hateren.15 These images
M. S. Keil and G. Cristobal Vol. 17, No. 4 /April 2000 /J. Opt. Soc. Am. A 699
each have an original size of 1536 3 1024 pixels. Theypossess an angular resolution of approximatively 1 arcmin per pixel, which yields a maximum spatial-frequencyresolution of 30 cycles per degree (c/deg) (the human eyecan resolve up to approximately 60 c/deg).16 They werecalibrated according to the procedure outlined in Ref. 17.Therein can also be found a more detailed description oftheir properties.
As we proceeded with the calculation of the mean ES,we discovered an artifact common to the images. Itcomes to light principally in the form of two broken verti-cal lines at high spatial frequencies (Fig. 2 below). Thiseffect might have been generated through electrical inter-ferences in the camera18 and can be seen only in the loga-rithmic ES (i.e., it is very small). We verified that thiseffect had no significant influence on our results.
We established six categories for the images, and a sub-set of van Hateren’s image database was selected andclassified accordingly. In what follows we will frequentlymake use of the contraction for each class, which will bedenoted by italic typeface.
The class city contains urban scenes. These show astrong prevalence of manufactured objects (e.g., houses,streets) compared with natural objects (such as trees). Ifa photograph is taken of such scenes, one typically ob-serves that the vertical edges of manufactured objects areparallel to the image frame. In other words, the horizonline (the line of gravity) in those images typically is par-allel to the horizontal (vertical) bounds of the image. Inthis case we are speaking about (strong) alignment. Thereason is that humans, while taking such photographs ina straight direction of gaze, usually tend to align theircameras. From this all images taken accordingly can beconsidered well aligned, although we did not explicitly re-sort to, for example, a carpenter’s level. Closer to natureare the scenes represented by a man. Here we find im-ages containing relatively few manufactured objects com-pared with natural ones. Typically, in these scenes therecan be seen a building hidden to some extent behindtrees, or streets and fences embedded in a countryside.The frequency of manufactured objects is considerablyless than that in city. As in city, the alignment in man isgood in general. The images in the category natural con-tain no obvious manufactured objects. Instead, we findin natural mainly scenes from inside the woods and thena few scenes from open landscapes with meadows, trees,and seas. There are also scenes that were taken by pho-tographing upward, and consequently this class containssome poorly aligned images as well. Comparing with cityor man, we can consequently expect in natural a biggervariation of the alignment. With straight we establisheda category of a selected subset of images of natural con-taining only scenes that were subjectively well aligned.The selected scenes in straight are further biased in thesense that they consist of a large number of scenes frominside the woods. In grass we find images that mainlyshow grass or flowers. There were also included somemacrophotographs of flowers in this class. Moreover, ingrass we can find many pictures taken from upside ratherthan from a perspective corresponding to that of a smallanimal. Consequently, grass consists of both well-aligned images and some for which alignment cannot be
defined (i.e., those that correspond to a straight view ontothe ground). Finally, nine pictures showing pure skywere examined. Alignment in this case is not defined,because these images show only one or a few clouds.
The number of pictures in each class were 121 for city,188 for man, 241 for natural, 56 for straight, 49 for grass,and 9 for sky.
3. STATISTICAL MEASUREMENTSA. Calculation of the Energy SpectraFor all measurements the original image size of 1024rows 3 1536 columns was reduced to a square image con-sisting of 1024 3 1024 pixels to perform an optimal fastFourier transform. The resizing was made such that thefirst 256 and the last 256 columns were dropped. Subse-quently, each image was normalized to 1 and weighted bya Gaussian with s 5 1024/4& centered in the middle ofthe image. In this way the window effect of the Fouriertransform is weakened. The windowed image was sub-sequently transformed into the spatial-frequency domain,and the ES was calculated as the square of the amplitudespectrum. The first row and the first column weredropped to achieve correct centering. In this representa-tion all energy values E(k) corresponding to the samespatial frequency k 5 iki are lying on a circle with radiusk. Then the average two-dimensional ES over all imagesin a category was calculated. Rather than relying ononly the original ES, we experimented with both a loga-rithmic and a compensated ES. The latter was calcu-lated by multiplying the ES by k2 isotropically (i.e., a cir-cular window function). The motivation for doing so wasthe fact that the energy of natural images is approxi-mately proportional to 1/k2 (see, e.g., Refs. 14 and 19).Thus, after this function is applied, the energy can be ex-pected to be roughly independent of spatial frequency(i.e., constant along the line of radius).
B. Calculation of Oriented Energy: Club IntegrationSchemeFrom here on we operate in the spatial-frequency domain.The OE was calculated by using what we have named theclub integration scheme (CIS), because the integrationarea at a certain angle of orientation resembles a club, ascan be seen in Fig. 1(a). The total integration area cov-ered is shown in Fig. 1(b). The clubs were shaped in bothangular and radial direction by using the power of a co-sine, i.e., cos2a U. U consists of a phase depending onspatial frequency and the angle of rotation f of the club.In the radial direction, which corresponds to the fre-quency bandwidth, a [ ak 5 4 was chosen. In the an-gular direction, which corresponds to the orientation se-lectivity, a [ aU 5 400 was chosen, which makes theclub rather slim. The peak value of the cosine was speci-fied to be at a frequency of approximately k 5 18 c/deg.In this way the intermediate frequencies exert a domi-nant influence on the results. Hence the club served as aweighting scheme for the underlying energies. We alsoexperimented with unweighted clubs (i.e., which simplycut out a piece of the ES), and the outcomes were similar.Furthermore, the results do not depend critically on ei-ther of the selected parameter values. The integration
700 J. Opt. Soc. Am. A/Vol. 17, No. 4 /April 2000 M. S. Keil and G. Cristobal
Fig. 1. (a) Single club (representing a weighted integration area in the energy plane) shown at orientation f 5 0° (corresponding tovertical image features) and f 5 90° (corresponding to horizontal image features) and (b) full ‘‘club swing’’ in the spatial-frequency plane(optical representation). The effective image size is 1023 3 1023 pixels.
scheme was compensated for small sampling artifacts.These occur because the integration area at oblique orien-tations is bigger than that at angles corresponding to car-dinal orientations, because of small nonzero weights.The angular resolution was set to Df 5 5°. This yields,in total, 72 oriented energies Ea( f). To simplify nota-tion, we use the index a to refer to the specific settings ofboth ak and aU . By definition, the ES has even symme-try, i.e., it holds that Ea( f) 5 Ea( f 1 180°). By takingthis into account, we need to analyze only half of all ori-entations.
C. Measures on the Oriented EnergyLet Ea
( s)( f) be the Gaussian-smoothed OE
Ea~ s!~ f! 5 Ea~ c! ^
1
A2psexpF2
~ f 2 c!2
2s2 G , (1)
where the symbol ^ denotes convolution in angle c, f isthe orientation on which the Gaussian is centered, and sis the Gaussian’s standard deviation.
By that we can define the energy at cardinal orienta-tions as
Ecardinal 5 Ea~ sc!
~0° ! 1 Ea~ sc!
~90° ! (2)
and the energy at oblique orientations as
Eoblique 5 Ea~ so!
~45° ! 1 Ea~ so!
~135° !. (3)
The spread of the Gaussian at the oblique orientationsso was chosen to be twice the standard deviation of theGaussian at the cardinal orientations sc ; hence so5 2sc . By setting sc [ s, we can drop the indices fromsc and so .
Then the COR is defined as
COR~ s! 5Ecardinal 2 Eoblique
Ecardinal 1 Eoblique. (4)
Smoothing (the convolution with the Gaussian) is ap-plied because in this way information from adjoining ori-entations (i.e., adjacent to the angles corresponding to the
centers of the Gaussians) is made available, althoughwith less weight. As a further consequence, the COR isalso getting a more smooth or stable measure; the align-ment is never perfectly equal in each considered image.This is due to the fact that the cardinal orientations in theimage (i.e., line of horizon and line of gravity) are neverperfectly aligned with the CIS at the corresponding orien-tations (i.e., 90° and 0°, respectively). Thus we obtainsome sort of fluctuation around the cardinal and obliqueorientations from image to image, which in a certainsense can be considered noise. By applying smoothing,we can alleviate this effect to some extent.
The denominator provides a normalization, such that21 < COR < 11. A positive COR indicates that wefound more energy at cardinal (i.e., horizontal and verti-cal) orientations, and this is a measure of the strength ofthe oblique effect. If a COR value is located around zero,then the energy can be expected to be distributed more orless uniformly over the considered orientations, and therewill be no oblique effect. Similarly, if the COR is nega-tive, then there will be more energy at oblique orienta-tions rather than an oblique effect.
Now we have to specify the free parameter s. For allcalculations s 5 1.5 was chosen, which gives a reason-able trade-off between robustness and blurring the OE.To estimate when the COR is significant, we calculatedthe mean COR( s) for uniform-distributed noise. Foreach value of s, the corresponding COR( s) was averagedover 100,000 trials (i.e., noise samples). In this way weget an average or mean value (which should ideally bezero) and an associated absolute deviation (we chose theabsolute deviation rather than the standard deviation).The deviation defines a positive and a negative limitaround zero, which yields a range in which our values donot posses significance (within this range we cannot dis-criminate signal from noise). Consequently, if we laterobtain a COR that lies above or below this rangeDCORcrit , this must be due to a property of the data dif-ferent from noise. Only in this situation can we considerthe COR significant, and we can be sure that the cardinal
M. S. Keil and G. Cristobal Vol. 17, No. 4 /April 2000 /J. Opt. Soc. Am. A 701
orientations indeed contain more energy than the obliqueones (or vice versa). For s 5 1.5 the COR range is de-fined as DCORcrit ª $CORu20.089 < COR < 0.089%.
We also experimented with different values of s andother COR definitions. The main effect in both cases wasa shift of the considered COR values by an offset value.The shape of the COR plots (i.e., COR versus spatial fre-quency) remained similar.
To measure the anisotropy of the energy at cardinalorientations, we define the AER analogously to the COR:
AER~ s! 5Ea~0° ! 2 Ea~90° !
Ea~0° ! 1 Ea~90° !. (5)
This measure gives the difference in energy corre-sponding to vertical (0° club orientation) and horizontal(90° club orientation) image features, and the denomina-tor provides a normalization. Thus it holds that 21< AER < 1. Negative values of the AER indicate a rela-tive preponderance of horizontal image features in therange of frequencies under consideration. The oppositeholds true for vertical image features, indicated by posi-tive values. As above, s 5 1.5 was chosen. This yieldsDAERcrit ª $AERu20.151 < AER < 0.151%. To calculatethe dependencies on spatial frequencies of AER and COR,we set the a values to aU 5 ak 5 150. This yields small,isotropic, bloblike integration areas. The integration ar-eas at different frequencies are simply rescaled versionsof each other. The frequency interval of [8,508] cyclesper image is passed through (or sampled) linearly with aresolution of Dk 5 5 cycles per image. As already men-tioned in Section 1, the specific values for aU and ak werechosen such that a good resolution is obtained, ratherthan setting them to values corresponding to the classicalreceptive fields of simple cells. For example, if we usedan octave scheme for sampling the frequency axis,14 wewould not get a sufficiently smooth plot that would allowus to see the frequency dependence of the COR or theAER with a good resolution along the spatial-frequencyaxis.
4. RESULTS AND DISCUSSIONThe logarithmic averaged power spectra for each classcan be seen in Fig. 2. A pseudocolor map was applied forpurposes of visualization. Figure 2(a) shows the meanES of city. There is a clearly visible strong concentrationof energy along the cardinal axes, and the shape of thedistribution resembles a diamond. Note that the influ-ence of perspective can be seen best in this class. Per-spective affects, in the first place, horizontal lines, whichpresent different angles when projected onto a two-dimensional plane. This leads to a broader distributionof horizontal image features, and for this reason energy isdiffusing away from the vertical axis in the ES. The en-ergy around the horizontal ES axis (vertical image fea-tures) is clearly more concentrated. This can be betterseen in Fig. 3(a). Here the OE is presented. The distri-bution around the orientation of 90° (vertical club; seeFig. 1), corresponding to horizontal image features, isagain much broader. Nevertheless, all distributions hereare very marked and show sharp peaks, as a result of a
prevalence of rectangular objects (e.g., buildings) overnatural ones. The dependence of the OE on spatial fre-quency can be seen in Fig. 4(a). This plot suggests anearly stable alignment over the considered frequencyrange, which is confirmed by examination of the COR plotin Fig. 5(a). The decrement of the degree of alignment athigher spatial frequencies at approximately 15 c/degseems to be a characteristic property of the classes man,natural, and straight. Since the falling off can be ob-served in all classes, this might indicate that alignmentgets weaker in general as the spatial frequency increases.This means that if we look at objects on a fine scale, wewill see that the associated features (lines, edges) occur atother than the cardinal orientations with higher probabil-ity. In the AER plot for city [Fig. 5(b)], the energy isroughly distributed equally on the cardinal axes, al-though with a slight bias in favor of vertical image fea-tures, in the range of low spatial frequencies. A curiosityin the shape of a marked peak can be found in the OE plotin Fig. 4(a) near the orientation of 90° and at a frequencyroughly near 17.5 c/deg (because of the three-dimensionalpresentation, this appears to be at approximately 20c/deg). Interestingly, this peak is also present in man atthe same orientation and at the same spatial frequency.Thus, because it is not present in any of the image setsthat contain only natural images, there must be sometypical manufactured feature that is shared by both thesets city and man. Of course, this is not easy to trackdown. Note that there is also another peculiar peak (butsmaller) in Fig. 4(b) in the 0°/180° orientation in man atapproximately 17 c/deg.
The ES of man [Fig. 2(b)] resembles again a diamond,but the energy at high spatial frequencies seems to bemore poorly concentrated at the cardinal axes than it is incity. This indicates that features (lines, edges) corre-sponding to high spatial frequencies are distributed moreuniformly over all orientations. Nevertheless, the ES ofman shows once again distinct peaks in Fig. 3(b), al-though the presence of natural features introduces now abroadening of the energy distribution around both cardi-nal orientations. Furthermore, if we compare the ES ofman and natural [Figs. 2(b) and 2(c), respectively], we ob-serve that adding manufactured object(s) to a naturalscene obviously increases the energy along the cardinalaxes over the whole range of frequencies. The markedconcentration of energy along the cardinal orientations inman along nearly the whole range of spatial frequencies(despite a relative preponderance of natural objects overthe manufactured ones) can be verified in Figs. 4(b) and5(c). Interestingly, the AER plot [Fig. 5(d)] of man (asopposed to city) indicates now a significant probability offinding more vertically oriented image features in therange of low spatial frequencies.
The ES of natural is roughly circular [Fig. 2(c)], al-though with slight bumps, which are the remains of thecorners of the diamond. One can, if at all, see a weakvertical line at lower frequencies in the ES. But inspec-tion of the OE in Fig. 3(c) clearly shows peaks, eventhough the energy is distributed very broadly around theorientation of 90°/270°. To understand this better, weconsider the frequency dependence of the OE in Fig. 4(c)as well as the COR [Fig. 5(e), solid curve] and the AER
702 J. Opt. Soc. Am. A/Vol. 17, No. 4 /April 2000 M. S. Keil and G. Cristobal
Fig. 2. Logarithmic average power spectra for each class in a pseudocolor representation hue–saturation–value color model. The cor-responding classes are (a) city, (b) man, (c) natural, (d) straight, (e) grass, and (f ) sky.
[Fig. 5(f ), lower solid curve]. Obviously, we are facedwith strong alignment at low spatial frequencies, whichfalls off by roughly 60% compared with its maximumvalue at high spatial frequencies. Another conspicuousfeature is the sign reversal of the AER at around 15 c/deg.The consistent interpretation of this effect is that the lowspatial frequencies are more strongly influenced by verti-cal aligned objects, whereas at high spatial frequencies
horizontal features predominate. Let us consider againFig. 4(c). Note that the image features with horizontalorientation have roughly the same energy over the wholerange of frequencies, while vertical image features loseenergy at the same time. Therefore the sign reversal ofthe AER is probably caused by a disappearance of verti-cally oriented image features at higher spatial frequen-cies. We cannot rule out that this may be a peculiarity of
M. S. Keil and G. Cristobal Vol. 17, No. 4 /April 2000 /J. Opt. Soc. Am. A 703
the scenes that we used. A consistent explanation, how-ever, would be that tree trunks at short distances signifi-cantly influence the lower spatial frequencies. With in-creasing viewing distance in a scene, the trunks aregetting more and more invisible. This is because theyare getting covered by branches and trunks from treessituated in the foreground, and so the probability of see-
ing a tree trunk that is far away is relatively low. At thesame time, the branches can be considered as deliveringthe main contribution for the energy around the horizon-tal orientations. This holds, as suggested by Fig. 4(c),over the whole range of frequencies. Moreover, becausetrunks and branches hide themselves mutually, addi-tional spatial frequencies are generated. Although this
Fig. 3. Oriented energy (OE) as calculated by the club integration scheme (CIS) by using the direct (i.e., original) averaged energyspectrum (ES). The corresponding classes are (a) city, (b) man, (c) natural, (d) straight, (e) grass, and (f ) sky.
704 J. Opt. Soc. Am. A/Vol. 17, No. 4 /April 2000 M. S. Keil and G. Cristobal
Fig. 4. Frequency dependence of the OE calculated from the mean power spectrum (ES) of each class. This was done by using the CISwith isotropic bloblike integration areas. The frequency axis was sampled linearly. Blobs at different frequencies were simply rescaledversions of each other (with bigger blobs at higher spatial frequencies). For reasons of visualization, the energy axis has been rescaledwith a factor proportional to the square of the spatial frequency. The corresponding classes are (a) city, (b) man, (c) natural, (d) straight,(e) grass, and (f ) sky.
M. S. Keil and G. Cristobal Vol. 17, No. 4 /April 2000 /J. Opt. Soc. Am. A 705
Fig. 5. Frequency dependence (part 1) of the cardinal versus oblique energy ratio (COR) (left column) and the aligned energy ratio(AER) (right column), calculated from the frequency-dependent OE as shown in Fig. 4. Note that the results for natural and straight aresummarized in one plot. In (e) the dotted curve shows the COR for straight and the solid curve the COR for natural. The dashedhorizontal lines refer to the mean value of ^AER( s 5 1.5)&, as obtained from uniform-distributed noise. The dotted horizontal lines arethe associated absolute deviations. Note the sign reversal in (f ). Corresponding classes are (a), (b) city, (c), (d) man, and (e), (f ) naturaland straight.
706 J. Opt. Soc. Am. A/Vol. 17, No. 4 /April 2000 M. S. Keil and G. Cristobal
may happen with equal probability at all orientations, itmight in this case especially add additional energy to thehorizontal orientations. However, this is speculation andshould be verified explicitly. By all means it would be in-teresting to see whether this behavior of the AER [Fig.5(f )] is mirrored in the primary visual cortex, too. If thisis the case, then we will find a relatively bigger number ofcells preferring vertical orientations to horizontal ones atlow spatial frequencies, whereas the opposite holds truefor high spatial frequencies. At intermediate frequencies(around 10–15 c/deg), the respective number of cellsshould be more or less equal. Furthermore, if we takeinto account that the human visual system can resolve upto 60 c/deg,16 it would also be interesting to see whetherthe decrement of the AER with increasing frequency iscontinued up to 60 c/deg or whether there is another in-flection point beyond 30 c/deg, which is the maximum spa-tial frequency examined here.
Another source of energy at high spatial frequencies isgrass, thickets, or shrubbery, which is present to a certainextent in all images of man, natural, straight, and grass.By inspecting Figs. 3(e) and 4(e) for grass, we can see thatgrass, thickets, and shrubbery also occur with some prob-ability at vertical orientations, although there seems to beno preference for horizontal orientations, and the distri-
bution around the orientation of 0°/180° is relativelybroad. This is quite surprising, because the images ingrass are taken arbitrary with respect to alignment. De-spite this, the relatively high degree of alignment, as sug-gested by the COR plot in Fig. 6(a), is caused exclusivelyby the nonexistence of peaks at the horizontal orientation.This is reflected by a high value of the AER (near 1) inFig. 6(b) and can also be seen in Fig. 4(e). Moreover, inFig. 4(e) we see that the energy at low spatial frequenciesis very noisy, which indicates the absence of a preferredorientation at low frequencies. Note that in Fig. 2(e) theES for grass shows characteristic rays emerging from thedisk around the center. These fine rays correspond, ofcourse, to some misaligned high-frequency features (i.e.,at other than oblique orientations). This indicates thatin grass the feature orientations at high spatial frequen-cies are distributed more uniformly over the angularscale—a clue to poor alignment of the individual imagesand perhaps also to the small number of images (49) usedin this class.
With straight is demonstrated what will happen if wepick out those images of natural that seem to be subjec-tively (i.e., to a human observer) well aligned. The prin-cipal difference with natural is an increase of the 0°/180°peak relative to the peak at 90°/270° [Figs. 3(c) and 3(d)
Fig. 6. Frequency dependence (part 2) of the COR (left column) and the AER (right column). See also the caption to Fig. 5. Corre-sponding classes are (a), (b) grass and (c), (d) sky. In (c) the region enclosed by the two dotted lines corresponds to DCORcrit .
M. S. Keil and G. Cristobal Vol. 17, No. 4 /April 2000 /J. Opt. Soc. Am. A 707
Fig. 7. (a) Cardinal vs. oblique energy ratio COR and (b) AER calculated from the mean ES for each class. Triangles: original ES,squares: k2-windowed OE, circles: logarithmic ES. See also the caption to Fig. 5.
and Figs. 4(c) and 4(d)]. Note also that in Fig. 3(d) weobtained a more rugged plot as a result of the smallernumber of images in straight (241 in natural as against56 in straight). In Fig. 5(e) the COR is presented fornatural and straight in one plot. In Fig. 5(f ) the AER isplotted for both classes. As can be easily seen, the maineffect in both cases (i.e., AER and COR) is that the graphsfor natural are shifted upward (i.e., to higher values ofthe AER and the COR), while their shape seems to bemaintained at the same time. This is a hint that, al-though natural contains some poorly aligned scenes, inthe end this does not have a significant effect. Thus the
Table 1. Number of Images FulfillingCOR − DCORcrit While Using the Raw ES (Direct),
effect of head rotation on the overall statistics seems to benegligible, given that we consider the displacement of theCOR as harmless in the sense that the relative number ofcells at cardinal and oblique orientations is preserved.Moreover, the same seems to hold for the AER. Here therelative number of cells dedicated to the processing ofhorizontal and vertical orientations seems to be pre-served, too. However, to clarify whether head rotation inany case amounts to merely a shift of the COR and theAER, an additional study should be undertaken, whichcalculates the COR and the AER for an image set as afunction of mixing well-aligned and poorly aligned im-ages, e.g., by systematically increasing the relativeamount of rotated versions of the original images (if ad-equate camera shots were not available).
In sky we examined only 9 images, so the results mustbe judged in light of this fact. As we can see in the ES ofsky [Fig. 2(f )], the energy of such images is concentratedat lower spatial frequencies in an isotropic fashion. Athigh spatial frequencies, only relatively small values ofenergy remain. For this reason the window artifact ofthe numerical Fourier transform becomes visible andleaves us with two spurious maxima in Fig. 3(f ) at cardi-nal orientations. If we had a lot of images in sky, we ide-ally would obtain no dependency of the energy on orien-tation, as long as we had not taken the images against thehorizon. Single clouds above our heads normally do notpossess a preferred orientation, whereas cloudy sky at thehorizon presents a strong horizontal component. How-ever, our images show only single clouds as seen aboveour heads.
Despite the window artifact of the Fourier transform,both the COR and the AER [Figs. 6(c) and 6(d), respec-tively] yield more or less reliable results. The COR liesonly in the noisy domain; that is to say, it cannot be con-sidered significant. And in consideration of this, theAER does not possess any meaning.
The results in terms of the COR and the AER as calcu-lated from the OE of Fig. 3 for each class are summarizedin Fig. 7. In addition, we demonstrated in these plots the
708 J. Opt. Soc. Am. A/Vol. 17, No. 4 /April 2000 M. S. Keil and G. Cristobal
effect of taking the logarithmic (circles) and thek2-windowed (squares) ES, respectively, as input for theCIS. The differences between the different spectra aresmall, and, more important, the monotonicity (in itsmathematical sense) is preserved. Thus, if the visualsystem makes a monotonic transformation of the lightpatterns impinging upon the retina, the statistical prop-erties of those patterns should be conserved and should bereflected in the visual cortices.
We also calculated the COR’s and the AER’s for eachindividual image. These results are summarized inTables 1 and 2, respectively. An extended version of thispaper, including a discussion of the results for the indi-vidual images, is available and can be requested from theauthors.
5. CONCLUSIONSIn this study we made a detailed examination of thealignment in natural scenes. As opposed to existingstudies,11–13 we quantified the degree of alignment (COR)and examined the dependence of this measure on spatialfrequencies. Our results for high frequencies confirmquantitatively the outcomes of the study from Coppolaet al.,7 although our images were not manually aligned.This suggests that (1) the oblique effect can be seen inmany kinds of natural images (city, man, natural), and inthis sense natural images share common statistical struc-tures that seem to be independent of the specific country-side in which they were gathered (Coppola et al. of courseused images showing different landscapes), and (2) align-ment can be produced, on the average, up to some certainamount of counteracting head movements. More impor-tant, we have seen that different sets of pictures generatealignment of different strengths. With straight we haveexemplarily shown how the alignment of natural changesif we consider only well-aligned images. Further on, wequantified the anisotropy of the energy between the (car-dinal) angles (0°/180° versus 90°/270°) by definition of theAER. This enabled us to examine the frequency depen-dence of the anisotropy, captured by the behavior of theAER.
Again, for high frequencies we obtained quantitativelythe same results as those of Coppola et al.,7 and thisagain may be a hint at another generic property of natu-ral images [see also Fig. 1(B) in the paper of van derSchaaf and van Hateren13]. Remarkably, in natural wefound a characteristic sign reversal of the AER [Fig. 5(f )],and its origin presumably lies in a covering scheme, i.e.,objects located in the foreground partially occlude back-ground objects (see Section 4).
To explain the higher probability of occurrence of fea-tures along the cardinal axes, Coppola et al. supposedthat ‘‘the reason for a bias toward the cardinal axes insuch different natural settings is presumably an omnipo-tent horizon dictated by the earth’s surface (which guar-antees horizontal components in most scenes) and anabundance of plants that use vertical support to counterthe force of gravity and horizontal extension to capturesunlight with maximum efficiency’’ (Ref. 12, p. 4005).However, in our natural image set those scenes showingopen landscapes (i.e., exposing a horizon) are clearly in
the minority (approximately 4%), and thus the horizoncannot account for the energy at horizontal orientationsin our results. Moreover, a straight line of horizonshould, on the contrary, produce narrower peaks at orien-tations of 90°/270°. This can be understood by consider-ing the power spectrum of a step function, which is typi-cally sharply localized. The only candidates that remainin this range of spatial frequencies are branches of trees,on the grounds of the specific images that we used.
Another remarkable finding is that there is a charac-teristic drop-off with higher frequencies of the COR, indi-cating that alignment gets less stable with increasingscale. It can be seen quite obviously in the classes city,man, natural, and straight. This behavior of the CORcannot be caused by the approximately 1/k2 proportional-ity of the energy found in natural images.14 We ruledthis out by considering not only the original ES but alsothe k2-windowed ES. The effect of using thek2-windowed ES on the COR (and the AER) is a shift ofthe curve. The shape of the curve is not altered criti-cally. The falling off cannot be due to perspective. Incity we have a preponderance of horizontal and verticalaligned features (lines, edges) because of manufacturedobjects. In this case perspective may be one source of thealignment’s decreasing stability as frequency increases.But perspective affects only the horizontal orientationand contributes equally to all frequencies. Furthermore,perspective leads to a diffusion of energy away from thehorizontal orientation. Consequently, the AER shouldindicate a relative preponderance of vertical image fea-tures. However, this is not the case. In city we have,apart from buildings, other objects with no preferredalignment, such as humans, cars, bicycles, shrubbery, etc.Typically these objects are in front of buildings, therebygenerating new spatial frequencies without any orienta-tion preference by means of occlusion of the background.Because the objects that occlude are typically muchsmaller than the background objects (such as a man infront of a house), the additionally introduced spatial fre-quencies should be intermediate to high, and this couldhelp to explain the behavior of the COR in city and man.A similar explanation can also be employed for natural,and in Section 4 we described the respective mechanism.
We are now trying to clarify whether the oblique effecthas its origin in phylogeny or ontogeny. If we take thephytogenetic point of view as being the origin of the ob-lique effect (oblique effect being due to evolution), thenthe classes city and man must be excluded from this dis-cussion. The first reason is, of course, that those types ofscenes are not natural, because of the presence of manu-factured objects. Or, in other words, those scenes occurtoo infrequently as opposed to natural scenes to provide arepresentative environment for an evolving organism.The second reason is that the scenes contained in city donot harmonize with what has been found neurophysiologi-cally. To be more precise, city does not show a significantanisotropy at cardinal orientations over the whole rangeof frequencies [see the corresponding AER plot in Fig.5(b)], although it shows a good alignment at the sametime [COR plot in Fig. 5(a)]. As mentioned in Section 1,Coppola et al.7 reported that, on the average, approxi-mately 2% more cortical area preferred horizontal to ver-
M. S. Keil and G. Cristobal Vol. 17, No. 4 /April 2000 /J. Opt. Soc. Am. A 709
tical orientations. In man we have a significant align-ment up to 10 cycles per degree but no significantalignment beyond this frequency [AER plot in Fig. 5(d)].
However, natural shows a corresponding significantanisotropy, with more energy at the vertical orientationat lower spatial frequencies and more energy at the hori-zontal orientation at higher spatial frequencies.
In our view it is unlikely that visual experience alonecan account to establish the neuronal substrate for the ob-lique effect. First evidence comes from recent neuro-physiological studies. Chapman and Bonhoeffer6 exam-ined eight developing and three adult ferrets. In allanimals they observed that more cortical area is dedi-cated to processing of the cardinal orientations. Further-more, they reported that the individual animals show agreat variability in the expression of this effect. This ob-lique effect could be seen before the onset of the criticalperiod of cortical plasticity, where visual experience getsimportant. The onset of the critical period in ferrets canbe estimated by adding two or three weeks to the begin-ning of the cat’s critical period of susceptibility, which be-gins at postnatal day 21 (see, e.g., Refs. 6 and 20 and ref-erences therein). Furthermore, all ferrets in theChapman and Bonhoeffer6 study were raised in the samevisual environment.
But for what reason then does visual experience count?A possible answer to this question is provided by Crairet al.20 They studied the impact of visual experience onorientation and ocular dominance maps by comparingnormal with binocularly deprived cats. Up to postnatalday 21, the maps of normal and deprived cats were indis-tinguishable from one another. However, from day 21on, the maps in the binocularly deprived cats begin to de-teriorate. Moreover, much stronger and more selectiveresponses were elicited up to postnatal day 21 by stimu-lation of the contralateral eye compared with that of theipsilateral eye. They concluded that pattern vision is re-quired for the maintenance of orientation selectivityrather than for the initial development. Thus there isevidence that visual experience might serve to establishthe responsiveness to the ipsilateral eye in a map that ini-tially is dominated by the contralateral eye. Genes seemto code for an orientation map, and segregation of whichinto ocular dominance columns is accomplished somehowby visual experience.
But do nonoptimally driven cells get unresponsive orare they changing their orientation preference? A studyof Sengpiel et al.5 provides evidence that cells are retunedrather than switched off. Moreover, with this resultSengpiel et al. questioned the notion of maintenance of re-sponsiveness. A prerequisite in any case is the plasticityof the underlying neural circuits. And it would not besurprising if these local cortical networks mediate bothocular dominance and orientation selectivity. Hence itcan be argued that one cannot switch off the plasticity fororientation column circuits while keeping it on for theocular dominance circuits at the same time, if we assumethat they are functionally separated at all. This may bea possible explanation of why visual deprivation in thecritical period of susceptibility leads to deterioration ofboth orientation selectivity and ocular dominance col-umns. Note that a consequence of taking this viewpoint
is that visual experience can only redistribute poorly usedorientation columns rather than creating them from somekind of tabula rasa state.
Let us now reconsider our results and compare the out-comes for the natural sets of images represented by thecategories natural, straight, grass, and sky. Let us as-sume that the oblique effect provides an increase in theevolutionary fitness of an organism. We have seen thatover the different sets of images the expression of the ob-lique effect differed quite strongly. If we now rely on ap-proximately one week of visual experience to establish thecortical orientation maps, then we will need to acquireduring this time the representative statistics through theset of images perceived. The statistics must be represen-tative in the sense that they are sufficient to establish theoblique effect. For all that, this seems unlikely, becauseduring the critical period of cortical plasticity many mam-mals rely on their parents. Thus the environment inwhich they spend the time during the critical periodmight not have the statistical traits that are needed foran establishment of the oblique effect, although the sta-tistics might be sufficient to establish ocular dominancecolumns. How far head rotation or other movement in-duced by active behavior interacts with the establishmentof the oblique effect in the critical period is an issue thathas yet to be clarified more. By virtue of our results, wecan conjecture that this may lead to a decrement of thebigger number of cells at cardinal orientations, i.e., atrend to make the number of cells per orientation equal,so that, in the end, the oblique effect might be weakened.So one can conclude by saying that providing orientationmaps through genes (including the oblique effect) wouldprobably be a more secure way, because the experiencedvisual statistics in the cortical critical period might not al-ways capture those relevant details that afterward wouldprovide a potential increase in evolutionary fitness. Notethat if this is true, then not fulfilling the latterrequirements—that is, relying solely on visualexperience—could amount to a corresponding selectioncriterion for the evolutionary process. The correspondinggenes would die out, leaving those genes that code for asingle orientation map as a trade-off between securityand limited space for information storage.
ACKNOWLEDGMENTSMatthias S. Keil thanks Javier Portilla and Donald Gal-logly for fruitful discussions. This work was sponsoredby the European Union INCO-DC961646 (DG-III) AM-OVIP project and the MAS3-CT97-0122 (DG-XII) ADIACproject. See http://www.iv.optica.csic.es/projects/amovip.html for further information.
Address correspondence to Matthias S. Keil at the loca-tion on the title page or by e-mail, [email protected].
REFERENCES1. J. P. Jones and L. A. Palmer, ‘‘An evaluation of the two-
dimensional Gabor filter model of simple receptive fields incat striate cortex,’’ J. Neurophysiol. 58, 1233–1258 (1987).
2. S. Apelle, ‘‘Perception and discrimination as a function ofstimulus orientation: the oblique effect in man and ani-mals,’’ Psychol. Bull. 78, 226–278 (1972).
710 J. Opt. Soc. Am. A/Vol. 17, No. 4 /April 2000 M. S. Keil and G. Cristobal
3. I. P. Howard, Human Visual Orientation (Wiley, New York,1982).
4. B. A. Olshausen and D. J. Field, ‘‘Wavelet-like receptivefields emerge from a network that learns sparse codes fornatural images,’’ Nature (London) 381, 607–609 (1996).
5. F. Sengpiel, P. Stawinski, and T. Bonhoeffer, ‘‘Influence ofexperience on orientation maps in cat visual cortex,’’ Na-ture Neurosci. 2, 727–732 (1999).
6. B. Chapman and T. Bonhoeffer, ‘‘Overrepresentation ofhorizontal and vertical orientations preferences in develop-ing ferret area 17,’’ Proc. Natl. Acad. Sci. USA 95, 2609–2614 (1998).
7. D. Coppola, L. White, D. Fitzpatrick, and D. Purves, ‘‘Un-equal representation of cardinal and oblique contours inferret visual cortex,’’ Proc. Natl. Acad. Sci. USA 95, 2621–2623 (1998).
8. J. D. Schall, D. J. Vitek, and A. G. Leventha, ‘‘Retinal con-straints on orientation specificity in cat visual cortex,’’ J.Neurosci. 6, 823–836 (1986).
9. D. C. Linden, R. W. Guillery, and J. Cucchiaro, ‘‘The dorsallateral geniculate nucleus of the normal ferret and its post-natal development,’’ J. Comp. Neurol. 203, 189–211 (1981).
10. M. I. Law, K. R. Zahs, and M. P. Stryker, ‘‘Organization ofthe primary visual cortex (area 17) in the ferret,’’ J. Comp.Neurol. Power 278, 157–180 (1988).
11. E. Switkes, M. J. Mayer, and J. A. Sloan, ‘‘Spatial frequency
analysis of the visual environment: anisotropy and thecarpentered environment hypothesis,’’ Vision Res. 18,1393–1399 (1978).
12. D. Coppola, H. Purves, A. McCoy, and D. Purves, ‘‘The dis-tribution of oriented contours in the real world,’’ Proc. Natl.Acad. Sci. USA 95, 4002–4006 (1998).
13. A. van der Schaaf and J. H. van Hateren, ‘‘Modeling thepower spectra of natural images: statistics and informa-tion,’’ Vision Res. 36, 2759–2770 (1996).
14. D. J. Field, ‘‘Relations between the statistics of natural im-ages and the response properties of cortical cells,’’ J. Opt.Soc. Am. A 14, 2379–2394 (1987).
15. http://hlab.phys.rug.nl/archive.html.16. F. W. Campbell and D. G. Green, ‘‘Optical and retinal fac-
tors affecting visual resolution,’’ J. Physiol. (London) 181,576–593 (1965).
17. J. H. van Hateren and A. van der Schaaf, ‘‘Independentcomponent filters of natural images compared with simplecells in primary visual cortex,’’ Proc. R. Soc. London Ser. B265, 359–366 (1998).
18. http://hlab.phys.rug.nl/imlib/peaks.html.19. D. L. Ruderman, ‘‘Statistics of natural images: scaling in
the woods,’’ Phys. Rev. Lett. 73, 814–817 (1994).20. M. C. Crair, D. C. Gillespie, and M. P. Stryker, ‘‘The role of
visual experience in the development of columns in cat vi-sual cortex,’’ Science 279, 566–570 (1998).
