Vol. 7, No. 12/December 1990/J. Opt. Soc. Am. A 2245
Ideal observer and absolute efficiency of detecting mirrorsymmetry in random images
Markku Tapiovaara
The Finnish Centre for Radiation and Nuclear Safety, P.O. Box 268, 00101 Helsinki, Finland
Received March 27, 1990; accepted July 18, 1990
The problem of detecting symmetry has been studied by using digitally generated images with random pixel values.The statistical efficiency of humans and a computerized observer, the cross correlator of the image halves, has beenevaluated. The efficiency of humans is approximately 100% when the image comprises only a few pixels and isnotably better than that of the cross correlator. When the number of pixels in the image is increased, thedetectability of symmetry gets better. For human observers detectability saturates, however, on a level correspond-ing to a modest number of pixels. Human efficiency in detecting symmetry is thus low when the image matrix size islarge.
INTRODUCTION
Images are often used for detecting the presence of a signalor for discriminating between signals in an image. Theseobservations are then used for decisions regarding the stateof the imaged object, as in medical diagnoses or applicationsof pattern recognition. The accuracy of the diagnosis, or thecorrectness of the decision, depends not only on the informa-tion present in the image but also on the observer's a prioriinformation regarding the detection task and the ability ofthe observer to use the available information." 2
Statistical decision theory provides concepts and tools forevaluating this information and the ability of the observer touse it. One of these is the ideal observer (e.g., see Refs. 2 and3) that is able to use the information in an optimal way for itsdecision. Its performance sets the theoretical best-possiblelimit on detection and in this respect can be interpreted as ameasure of the useful information available. Real observ-ers, whether humans, neural networks, or other pattern-recognition algorithms, do not generally have this ability,and their error rates in detection are larger than the theoret-ical limit.
An important quantity here is observer efficiency, or sta-tistical efficiency, 2 4 5 which shows the fraction of availableinformation that the real observer is able to use in makingdecisions. Measuring this efficiency gives insight, for exam-ple, into questions of potential benefits of image manipula-tion or enhancement methods in the case of human observ-ers6 or the potential for developing more accurate pattern-recognition methods in automatic detection systems.7' 8
Strictly, the concept of observer efficiency is rigorous onlyunder certain conditions, which will be considered in thenext section, but can be used as a useful approximation inother situations as well.
The most common detection task that has been consid-ered is that of detecting a specified object embedded instationary Gaussian noise. This task is often referred to as asignal-known-exactly (SKE) task, and in this task the ob-server is provided access to all possible information regard-ing the signal, the background, and the noise properties,excluding only information of whether the signal is present
in the image. The ideal-observer detection strategy in aSKE task is well known and simple.'-3 It can be implement-ed by cross correlating the (prewhitened) image data withthe expected signal or, equivalently, by matched filtering the(prewhitened) image data. Human-observer efficiencies ofthe order of 50% (up to 70%) have been found in these tasksunder suitable conditions.9"10 This has been considered tosuggest that human visual processing resembles the opera-tions of the ideal observer on the image data, although hu-mans seem to lack the ability to prewhiten the datall 2 or tohave access to the data only through frequency-selectivechannels.'3
If the access of the observer is limited to less than full apriori knowledge regarding the detection task, for example,if a range of possible signals or backgrounds is given, theerror rate of the observer will get larger because some of theinformation that it could have benefited from is unavailable.This is a more realistic situation than the simple SKE casebut generally causes the mathematical description of theideal observer to become complicated. Human efficiency insuch tasks has been investigated by Burgess6"14 and Burgessand Ghandeharian.'5 "16 In these studies it was shown thatuncertainty regarding signal parameters generally did notimpair human efficiency compared with that in SKE tasks;the degrading effect on ideal-observer performance wasequal. The inclusion of a structured backgrounds was seento reduce human efficiency, however.
Discriminating between correlations in an image is anoth-er type of visual task with signal uncertainty. Wagner etal.' 7"8 have studied human ability to detect jittery gridsembedded in noise textures simulating B-scan ultrasoundimages. They compared human performance with that of anonideal algorithm and found that human efficiency is lowin this task, only 5-10% of the efficiency of the algorithm,which demonstrated human inability to detect efficientlydifferences in such correlations.
A related visual task, for which there is no particularsignal shape to be detected but the task is to detect a moreabstract property of the image, mirror symmetry, has beenstudied by Barlow and Reeves.'9 They made tests usingrandom dot displays and found a high value of 25% for
2246 J. Opt. Soc. Am. A/Vol. 7, No. 12/December 1990
human efficiency by comparison with a counter of potentialsymmetric pairs, essentially a cross correlator of the imagehalves. In a later study Maloney et al.
2 0 studied the detect-ability of Glass patterns that also can be described as variouscorrelations in the image. Their results suggest that hu-mans are efficient in detecting these patterns as well, al-though no value for the efficiency was given.
This paper will focus on detecting mirror symmetry inrandom images that are analogous, although not identical, tothose considered by Barlow and Reeves.19 First the mea-sures of detectability used in this paper will be consideredbriefly, and the ideal detection algorithm will be found.Then some performance tests for human observers and acomputerized observer, the cross correlator, will be de-scribed.
ELEMENTS OF STATISTICAL DECISIONTHEORY
There are various definitions of optimal decision making, oroptimal detecting. These can be given in terms of receiveroperating characteristics3 by summary measures such asminimum-error probability, the Bayes criterion for mini-mum expected cost, and the Neyman-Pearson or minimaxcriterion.1"3 Minimum-error probability is a special case ofBayes criterion for equal cost of false positive and negativeerrors; the Bayes criterion in its more general form should beused in the case for which false positive and negative errorsare of different importance. The use of Neyman-Pearson orminimax criteria is reasonable when the a priori probabilityof signal presence is not known.
Given any of these criteria, it can be shown",3 that optimalperformance can be achieved by basing the decision on thelikelihood-ratio test or another decision function that ismonotonically related to it. It will also be assumed that realobservers analogously base their decision on the value ofsome function f(x) of the data, which will be called theirdecision function. (Human observers exhibit internalnoise'4; their f is a random function.)
When the conditional distributions of the value of thedecision function, given signal present or absent, are Gauss-ian with equal variances, the performance of the observer isspecified by the detectability index d', the difference of themeans of these two distributions divided by the square rootof their common variance'; d' can be understood as thesignal-to-noise ratio at the decision level.
If the conditional distributions are Gaussian but the vari-ances are not equal, d' does not alone specify observer per-formance in a yes-or-no detection test. However, if a two-alternative forced-choice (2-AFC) test3 is being used (forthis test the observer chooses the more likely of two images,one of which contains the signal and the other does not), thesymmetry of the problem simplifies the analysis, and a moregeneral d', defined as
determines the probability of correct answers in the 2-AFCtest as
p(correct) = (1/>2/r) J exp(-x2 /2)dx. (2)
In Eq. (1) ( ) have been used for expected value and o-2 forvariance. s and n denote the conditions of the signal's beingpresent and absent, respectively.
Observer efficiency, or statistical efficiency F0, can thenbe estimated from observer-performance studies by the ratioof the squares of do' and d, of the real and ideal observer,respectively 4 5:
F0 = (d0 '/d/ )2 . (3)
The above analysis is not strictly rigorous, if the distribu-tions involved are not Gaussian, but is nevertheless useful inevaluating performance. In practical observer-performancetests receiver-operating-characteristic curves can be fittedwell with the assumption that the distributions are normal.21This should then hold also for tests such as the 2-AFC test,while the fraction of correct answers in the 2-AFC test isequal to the area under the receiver-operating-characteristiccurve,' a commonly used nonparametric measure of detect-ability.
DESCRIPTION OF THE VISUAL TASK
Barlow and Reeves19 studied the efficiency of human observ-ers in detecting mirror symmetry in images consisting of agiven number of bright randomly positioned dots. The im-ages belonging to the symmetry class were generated byplacing M dots randomly in the image area, another set of Mdots in positions that were symmetrically located about themidline of the image, and N noise points randomly in theimage. The images for the nonsymmetric image class weregenerated by placing all the 2M + N dots randomly in theimage area. The task of the observer was to decide fromwhich of the parent populations, symmetric or nonsymmet-ric, the observed image was drawn.
It was clear that humans could not compare the positionsof the dots with an accuracy similar to the machine obser-ver's. In order to make the difficulty of the task morecomparable between humans and the machine observer,Barlow and Reeves reduced the accuracy with which the dotpairs were placed in the symmetric display, which removedthe extra advantage of the machine reader's use of accurateposition information.
Barlow and Reeves concluded that the optimal procedurefor discriminating the symmetric from the random patternswas to search the tolerance area corresponding to each dot tofind the total number of pairs that would qualify as deliber-ately generated pairs and to base the decision on this num-ber. By comparing human performance with that of thisprocedure they found an efficiency of 25%, when the toler-ance range of symmetric dot position was 12 min or more indisplays subtending 2 deg at the observer's eye. Barlow andReeves pointed out that this counter was not exactly ideal intheir test when the tolerance range of placing the dots waslarge. This suggests that the test be reformulated somewhatto make the ideal likelihood-ratio approach feasible.
An image consisting of randomly placed dots, the accuratepositions of which are not known, will be considered. In-stead of the dots themselves, only the total number of dots inany pixel area is shown, displayed as the brightness of thispixel. When the image consists of only a few (2k) sufficient-ly large pixels, the observer does not need to estimate posi-tions accurately. These human-observer tests were restrict-
Markku Tapiovaara
Vol. 7, No. 12/December 1990/J. Opt. Soc. Am. A 2247
SYMMETRY
AXISFig. 1. Example of the image matrix showing the division into twoimage halves and the indexing of the pixels. In this example thenumber of pixel pairs k is four.
ed to images with only a few different pixel values in order tosuppress the effect of the human's inability to discriminatebetween closely spaced gray levels. The accurately posi-tioned dot displays of Barlow and Reeves are identical tosuch images in the case of small pixels and a low density ofdots, except for the constraint on dot numbers in their dis-plays.
An example of such an image matrix, also depicting theindexing of the pixels, is shown in Fig. 1. Let us call theimage half consisting of the pixels on the left-hand side ofthe symmetry axis image 1, and that on the right-hand image2. In this example the number of pixels in either half-imagek is four.
THEORY
In what follows, vector quantities will be denoted by bold-face characters and random variables by script characters.
The pixel values Y,(i) and '2(i), where the subscripts 1and 2 refer to the left- and right-hand side image halves,respectively, were generated as follows. The signal wasgenerated by independently sampling pixel values l(i),i = 1, ... , k, for the pixels on the left-hand side of thesymmetry axis from a Poisson distribution with mean value
added to the image by sampling values nl(i) and n2 (i) forboth sides of the image from another Poisson distributionwith mean value n, to get the final pixel values Y,(i) and 52 (i)by
J11(i) = &(i) + nl(i),
52 (i) = 4i2(i) + n 2(i) (4)
It can be noted that this method of generating pixel values isequivalent to first sampling the total number of dots fromPoisson distributions with mean values ks and kn and plac-ing the dots randomly in the image halves.
Then, by forming k-dimensional vectors of the pixel val-ues and remembering that, according to this procedure ofgenerating the images, the random variables al(i), 4 2 (i),
n,(i), and n2 (i) are all independent in the case of nonsym-metric images, as are 41(i), ni(i), and n 2 (i) in the case ofsymmetric images, we get
Combining Eqs. (5)-(7), we obtain the likelihood ratio
(7)
P(S7 = I '2 = Isymmetric)-LI(Il, I2) = P(Jl = I 2 = I21nonsymmetric)
k rmin[Ii),12(i)] I3
= 14 1,= es m![I(i) -L()!12()!
- M]![12 (i)
Smn I(i)+I2(i)-2m 1- m]! (s + n)I(i)+I 2(i)
s. If the image was to represent a sample from the symmet-ric population, the same values were given also to the sym-metrically located pixels, 4 2 (i) = (i). If the image was torepresent a sample from the nonsymmetric population, thevalues for the pixels on the right-hand side of the axis weresampled from the same distribution. After this, noise was
which is our ideal-observer decision function in the case ofsignal and noise dot numbers being Poisson distributed.
It should be noted that the displays of Barlow andReeves'9 were not this simple. In their simplest case, wherethe dots were positioned accurately, the pixel values werenot independent because the numbers of signal and noise
(8)
I1(4) IO ) Ij.(1) 12.(4)
IO ) ii(2) I2.(2) IZ(3)
Markku Tapiovaara
(i = 1 .. , k).
2248 J. Opt. Soc. Am. A/Vol. 7, No. 12/December 1990
dots were given. In the case of inaccurately positioned dotsfurther dependency between neighboring pixel values is in-duced.
In the limit of sparse dots, s + n - 0, the probability of onepixel's being occupied by two or more dots is small, and it canbe shown that, in this limit,
L' = n(L) sk + ln[n/(s + n)] E [.,(i) + 72(i)]
+ ln[(s + n2 )/n2 1 E Y,(i) 2(i)j. (9)
The first sum in this approximation is the total number ofdots in the image, and the last sum is equal to the idealobserver suggested by Barlow and Reeves. This last sumcan also be seen to be similar to the common cross-correlat-ing operation in SKE tasks: the template here is one of theimage halves, the similarity of which is compared with theother half.
The performance of the ideal observer [Eq. (8)] could beevaluated if the distributions of fI under both of the parenthypotheses were known. The complexity of the expressionmakes this impractical. Therefore its performance has beenevaluated and compared with the cross correlator
- = .l * 12 = E .1(i)Y2(i) (10)
and human observers using actual test images. These ex-periments will be described in the next section.
METHODS
We designed our performance tests as 2-AFC tests for whichtwo images were shown at the same time to the observer andthe task was to decide which one of them was more likelydrawn from the symmetric parent population. This test waschosen in order to simplify the analysis of performance andto remove the need of the observer to maintain any specificcriterion for the signal's being present.3
The images were generated by using a computer (IBMPersonal Computer AT) equipped with a frame-grabberboard (PCVISION-plus, Imaging Technology Incorporat-ed). Each display consisted of two images, one below theother, one of which was generated according to the symmet-ric and the other according to the nonsymmetric methodexplained in the previous section. Which one of them wasshown as the upper image was determined by the program'suse of a random-number generator.22 The random-numbergenerator was also used for generating the pixel values bysampling the appropriate Poisson distributions for the sig-nal and the noise values for each pixel.23 The size of theimages displayed on the video monitor (Model 44BM, SaloraOy) was kept constant (two 6.2 cm X 9.3 cm images on themonitor screen) for all image matrix sizes by varying the sizeof displayed pixels. The relation between pixel value andbrightness was linear, with black corresponding to the lowestand white to the highest pixel value in the image. Thiscorrespondence was shown to the observer as a step wedge atthe edge of the monitor screen. Typically, the images con-sisted of only a few (2-10) gray levels, depending on thevalues of the Poisson parameters and the image matrix size.
(a) (b)
Fig. 2. Examples of displays belonging to the symmetric population. In these images the underlying Poisson distribution mean values are s =0.25, n = 0.25. The number of pixel pairs is (a) k = 4, (b) k = 100.
Markku Tapiovaara
Vol. 7, No. 12/December 1990/J. Opt. Soc. Am. A 2249
F1.0
0.6-
o.O:
0.2
nFig. 3. Efficiency of the cross correlator as a function of mean values s and n for the large image-matrix-size limit.
3
Table 1. Detectability Indices of the Ideal Observer (d1 '), Cross-Correlating Observer (dc'), and Human ObserverA (dh') for Various Image Parameter Valuesa
a The number of pixel pairs in the image k = 100. s and n are the mean values of the underlying Poisson distributions for the signal and noise contributions, re-spectively. The detectability indices have been estimated by using Eq. (2). Values in parentheses have been obtained by using Eqs. (12). The last column showshuman statistical efficiency Fh. Dashes indicate 100% correct test results; d' could not be estimated.
The borders of the images were displayed as gray lines andthe position of the symmetry axis as a white line. Examplesof displays are shown in Fig. 2.
The observer responded by using the keyboard of thecomputer, which kept a record of the responses. After eachresponse the observer was given the correct answer as feed-back. The viewing distance was not fixed; the observerswere allowed to view the images at any distance, which theychose to be 1-2 m. The time that the observer used for his orher decision was not restricted either and was typically a fewseconds per image pair, being short for small, and somewhatlonger for large, image matrices. Three human observerswere employed; below they will be identified as observers A(the author), B, and C. Observers A and C wore eyeglasses;with these corrections all observers were normal sighted.
Each image pair was also analyzed by two machine observ-ers with differing algorithms; one was the ideal observershown in Eq. (8), and the other the cross-correlating observ-er [Eq. (10)]. The responses of these algorithms were re-corded in the same data file as the correct answer and theresponse of the human observer. After each test series, 50-500 image pairs shown, the fraction p of correct answers foreach observer was obtained as
p = n/N -4 N = n/N 4 N (n-n t (11)
where n is the number of correct answers in N trials. Theerror shown corresponds to one standard deviation. d, df',and observer efficiency F0 were calculated from Eqs. (2) and(3).
Markku Tapiovaara
2250 J. Opt. Soc. Am. A/Vol. 7, No. 12/December 1990
2.
1.0
(a)
0/5
0.5
0.2
n1.0 - °0.03
1.5
1.5
-1.5
(b)
Fig. 4. Dependence of the detectability index d' on the mean values sand n for the large image-matrix-size limit: (a) ideal observer, (b) crdsscorrelator.
When the detection task is easy for the observer (d' large),it is difficult to obtain a reliable estimate of d' by actualtesting because incorrect answers are infrequent. For thepurpose of finding dZ and d' in such situations note thatboth the correlator and the logarithm of the ideal likelihood-ratio observer are sums of independent, similarly distribut-ed pixel-pair contributions, e, and e, respectively. In the
limit of a large number of pixels in the image the conditionaldistributions LX and n(Lj) are thus Gaussian, and we ob-tain
Vol. 7, No. 12/December 1990/J. Opt. Soc. Am. A 2251
d,= (( ilsymmetric) - (lnonsymmetric)
[o-2(elsymmetric) + Of (2Ilnonsymmetri )]1/2(12)
The mean values and variances needed in Eqs. (12) can becomputed with good accuracy and provide good estimates ofdi and d,' for images with sufficiently large matrix sizes.
RESULTS AND DISCUSSION
Tests were conducted for various image matrix sizes k rang-ing from 1 to 2500 and values of signal and noise Poissonmean values s and n ranging from 0.03 to 1.5.
As can be seen from Eqs. (12) the efficiency of the correla-tor Fc is independent of image matrix size in the limit of large
Table 2. Detectability Indices of the Ideal Observer (d'), Cross-Correlating ObserverA (dh') for Various Image Parameter Valuesa
a k is the number of pixel pairs in the image; s and n are the mean values of the underlying Poisson distributions for the signal and noise contributions,respectively. The detectability indices have been estimated by using Eq. (2). Values in parentheses have been obtained by using Eqs. (12). The last column showshuman statistical efficiency Fh. Dashes indicate 100% correct test results; d' could not be estimated.
A
1+7
3
2
1
0
1
4
I1 0 1 0 0
4
1 000 1 0 0 0
N U M B E R OF P I XEL PA I RS, kFig. 5. Detectability index d' of the observers as a function of image matrix size. k is the number of pixel pairs in the image, and theunderlying Poisson distribution mean values ares = 0.15, n = 0.35. The data points have been slightly displaced horizontally for clarity; actualvalues of k are 1, 4, 16, 100, 625, and 2500: solid squares, ideal observer; solid triangles, cross correlator; stars, human observer A; open circles,human observer B; diamonds, human observer C. The curves are computed using Eqs. (12): solid curve, ideal observer, dashed curve, crosscorrelator. Human observer data points have been obtained from N = 100-200 trials per datum, except N = 500 for observer A, k = 4; N = 50for observers B and C, k = 625, 2500. The error bars represent one standard deviation from the mean fraction of correct answers [Eq. (11)].
Markku Tapiovaara
I
2252 J. Opt. Soc. Am. A/Vol. 7, No. 12/December 1990
matrices. F is shown as a function of parameter values sand n in Fig. 3. The efficiency is high, although not unity,for low values of s and n, as can be expected based onexpression (9). The efficiency decreases as the values of theparameters are increased and is particularly sensitive to thevalue of parameter s.
Table 1 shows the d' obtained for the human observer A,and for the cross-correlating and the ideal observers at vari-ous mean values s and n, for a modest image matrix size k =100. The values of d' and d were calculated by usingactual test results. These values agree well with the valuescalculated by using Eqs. (12), shown in parentheses. The d'values of the human observer are comparable with thoseobtained by Barlow and Reeves,'9 as well as with those ob-tained by Maloney et al.
2 0 for detecting Glass patterns for asimilar signal-to-noise ratio.
The human d' appears to be an approximate function ofsignal-to-noise ratio in a wide range of these parameters, aswas seen also in Ref. 20 for the detection of Glass patterns.The more detailed dependence of d' and d,' on the meanvalues s and n is shown in Fig. 4.
Human-observer efficiency is of the order of 10% in theseexperiments with k = 100, except in the case of sparse dots (sand n small) for which values greater than 30% are obtained.These higher values can be associated with the modestamount of information present in the images; there are notmany occupied pixels for the ideal observer to analyze either.
Table 2 shows the d' obtained in tests with varying imagematrix size. In the test series with k = 1, s = 0.15, n = 0.35,human performance was slightly worse than pure guessing;the fraction of correct answers was 0.49. In the test serieswith k = 4, s = 0.15, n = 0.35 human performance was bychance better than that of the ideal observer; the human andthe ideal observers answered correctly 336 and 335 times,respectively, in 500 trials.
When the number of pixel pairs in the image is sufficientlysmall, human performance is better than that of the crosscorrelator. The data for s = 0.15, n = 0.35 are plotted as afunction of image matrix size in Fig. 5, showing the perfor-mance of all three human observers. The d' of the machineobservers increase continuously with image matrix size,whereas the d' of the human observers saturate to levelscorresponding to the information of only a few pixel pairs.Human efficiency is low when the image matrix size is largebut can be very high, approximately 100%, when the imagematrix size is small.
CONCLUSIONS
Based on the experience of SKE tasks one could imaginethat optimal performance in detecting symmetry could beachieved by considering one half the image as the signalmodel and by cross correlating the other half with this mod-el. At least in the task considered, however, this method isnot equivalent to the optimal; the ideal observer uses otherfeatures of the image, too.
When the mean number of dots per pixel is small theefficiency of the cross correlator can be greater than 80%, butfor larger mean numbers it decreases to low levels, less than5% for the parameter ranges in this study. The efficiency ofthe cross correlator is independent of image matrix size inthe limit of large image matrices.
Human efficiency of detecting symmetry is low when theimage matrix size is large; that is, when there are numerousindividual pieces of information to be considered. Whenthe number of pixel pairs in the image is sufficiently low,human efficiency approaches 100% and is notably betterthan that of the cross correlator. This indicates that simplecross correlating of the image halves cannot be the operationthat humans perform on the image data in our detectiontask. The saturation of the human d' at large image matrixsizes suggests, further, that humans are unable to use morethan a modest number of pixel pairs, or visual features corre-sponding to this information, in this task of detecting sym-metry.
Human visual efficiency in the (seemingly) more simpleSKE tasks has been found to be approximately 70% maxi-mally. An efficiency approaching 100%, as found in thisstudy, is surprising in the light of the versatility of humanvision. Ideal performance typically requires detectionmethods that are tailored exactly to the given detection task.If the task is modified, the performance of such a specializedobserver is usually severely impaired. Humans, however,are efficient observers in widely varying visual tasks.
The theory of ideal detection and performance, althoughdescribed in terms of images in this paper, is applicable toother kinds of data as well. The task of symmetry detection,or the analogous task of detecting similarity between twosets of random data, might be a useful test not only inpsychophysical studies but also in evaluating the data-utili-zation efficiency of general-purpose pattern-recognition al-gorithms, such as neural networks.
ACKNOWLEDGMENTS
The problem of detecting symmetry was pointed out to theauthor by Robert F. Wagner and Kyle J. Myers. Discus-sions with them have benefitted this study greatly.
REFERENCES
1. A. D. Whalen, Detection of Signals in Noise (Academic, NewYork, 1971).
2. R. F. Wagner and D. G. Brown, "Unified SNR analysis of medi-cal imaging systems," Phys. Med. Biol. 30, 489-518 (1985).
3. D. M. Green and J. A. Swets, Signal Detection Theory andPsychophysics (Wiley, New York, 1966).
4. W. P. Tanner and T. G. Birdsall, "Definitions of d' and n aspsychophysical measures," J. Acoust. Soc. Am. 30, 922-928(1958).
5. H. B. Barlow, "The efficiency of detecting changes in density inrandom dot patterns," Vision Res. 18, 637-650 (1978).
6. A. E. Burgess, "Statistical efficiency of perceptual decisions," inApplication of Optical Instrumentation in Medicine XII:Medical Image Production, Processing, Display, and Archiv-ing, R. H. Scheider and S. J. Dwyer III, eds., Proc. Soc. Photo-Opt. Instrum. Eng. 454, 18-26 (1984).
7. D. G. Brown and M. F. Insana, "Decision function efficiency forhigher order imaging tasks," in Medical Imaging II, R. H.Scheider and S. J. Dwyer III, eds., Proc. Soc. Photo-Opt. In-strum. Eng. 914, 76-79 (1988).
8. D. G. Brown, "Neural networks and ideal observers," in MedicalImaging III: Image Formation, S. J. Dwyer III, R. G. Jost, andR. H. Scheider, eds., Proc. Soc. Photo-Opt. Instrum. Eng. 1090,154-163 (1989).
9. A. E. Burgess, R. F. Wagner, R. J. Jennings, and H. B. Barlow,"Efficiency of human visual signal discrimination," Science 214,93-94 (1981).
10. A. E. Burgess, R. J. Jennings, and R. F. Wagner, "Statistical
Markku Tapiovaara
Vol. 7, No. 12/December 1990/J. Opt. Soc. Am. A 2253
efficiency-a measure of human visual signal detection perfor-mance," J. Appl. Photogr. Eng. 8, 76-78 (1982).
11. A. E. Burgess, R. F. Wagner, and R. J. Jennings, "Human signaldetection performance for noisy medical images," in Proceed-ings of the International Workshop on Physics and Engineer-ing in Medical Images, 0. Nalcioglu and J. M. S. Prewitt, eds.(Institute of Electrical and Electronics Engineers, New York,1982), pp. 99-105.
12. K. J. Myers, H. H. Barrett, M. C. Borgstrom, D. D. Patton, andG. W. Seeley, "Effect of noise correlation on detectability ofdisk signals in medical imaging," J. Opt. Soc. Am. A 2, 1752-1759 (1985).
13. K. J. Myers and H. H. Barrett, "Addition of a channel mecha-nism to the ideal-observer model," J. Opt. Soc. Am. A 4, 2447-2457 (1987).
14. A. E. Burgess, "Detection and identification efficiency: an up-date," in Application of Optical Instrumentation in MedicineXIII, R. H. Schneider and S. J. Dwyer III, eds., Proc. Soc.Photo-Opt. Instrum. Eng. 535, 50-56 (1985).
15. A. E. Burgess and H. Ghandeharian, "Visual signal detection I:ability to use phase information," J. Opt. Soc. Am. A 1, 900-905(1984).
16. A. E. Burgess and H. Ghandeharian, "Visual signal detection II:effect of signal-location uncertainty," J. Opt. Soc. Am. A 1,906-910 (1984).
17. R. F. Wagner, M. F. Insana, D. G. Brown, B. S. Garra, and R. J.Jennings, "Texture discrimination: radiologist, machine, andman," in Vision: Coding and Efficiency, C. Blakemore, ed.(Cambridge U. Press, to be published), Chap. 29.
18. R. F. Wagner, M. F. Insana, and D. G. Brown, "Progress insignal and texture discrimination in medical imaging," in Ap-plication of Optical Instrumentation in Medicine XIII, R. H.Schneider and S. J. Dwyer III, eds., Proc. Soc. Photo-Opt. In-strum. Eng. 535, 57-64 (1985).
19. H. B. Barlow and B. C. Reeves, "The versatility and absoluteefficiency of detecting mirror symmetry in random dot dis-plays," Vision Res. 19, 783-793 (1979).
20. R. K. Maloney, G. J. Mitchison, and H. B. Barlow, "Limit to thedetection of Glass patterns in the presence of noise," J. Opt. Soc.Am. A 4, 2336-2341 (1987).
21. J. A. Hanley, "The robustness of the 'binormal' assumptionsused in fitting ROC curves," Med. Decision Making 8, 197-203(1988).
22. J. C. Ehrhardt, "Generation of pseudorandom numbers," Med.Phys. 13, 240-241 (1986).
23. E. D. Cashwell and C. J. Everett, A Practical Manual on theMonte Carlo Method for Random Walk Problems (Pergamon,New York, 1959).
