Image and Vision Computing 14 (1996) 277-283 ELSEVIER Connectionist median filtering networks Andrew Hunter School of Computing and Information Systems. University of Sunderland. Priestman Building, Green Terrace, Sunderland SR2 7EE, UK Received 7 July 1995; revised 5 September 1995 Abstract A family of connectionist rank filtering networks is presented, suitable for real-time median filtering. It is demonstrated that these have low complexity (from 0(N2) processing elements and connections for a single output N element filter to O(N*) for a fully parailelised N x N filter). The networks have some unusual requirements in terms of processing elements and topology compared with the standard neural network models. The feasibility of VLSI implementation of the networks is discussed. Keywords: Median filtering; Connectionist networks; VLSI implementation 1. Introduction Median filtering is a widely used non-linear technique in image processing [1,2]. It belongs to the class of local operators, where the output value of a pixel is determined from input pixel values within a small neigh- bourhood of the output pixel (also referred to as the ‘window of the filter’). The neighbourhood is usually square, although straight and diagonal cross shapes are also used; it is invariably symmetric about a centre point. The median is calculated by ranking the input pixel values according to magnitude and selecting the middle value, i.e. for set S = {Si : 0 5 i < N}, me49 = Q((N-I)/z), where the rank R is defined by sR(i+ r) > SR(i), Y’i. As such, median filtering is an example of rank filtering [2], in which input pixels are ordered and the output pixel is set to the input pixel of a particular rank. Other important rank filters are the maximum and minimum filters, max(S) = sRCN_ ,), min(S) = sRCo). The useful properties of median filters have been widely studied [2-61. These include the smoothing of noise and the preservation of edge shapes. Median filters are considered particularly effective for the removal of impulse noise [5,6], which is usually caused by bit errors during data capture or transmission. This combination of properties makes median filtering suitable for pre-processing of images prior to the appli- cation of other techniques for image enhancement and segmentation. 0262~8856/96/$15.00 0 1996 Elsevier Science B.V. All rights reserved SSDZ 0262-X856(95)01068-8 The most significant drawback of the median filter is its computational complexity. Ndively, the median is produced by sorting the set of input values and selecting the middle ranked value. The quickest sorting algo- rithms have cost O(N log2 N) for N elements [7]. Various authors have suggested improved techniques for the implementation of median filters. Some of these [8,9] take advantage of the fact that as the filter window is moved across the input image, only some of the pixels change. A histogram or sorted list can be maintained, with old values being removed and new ones added at each move. This method is considerably faster than conventional algorithms, especially for larger windows. Another approach is to calculate the median by sorting the pixel values one bit at a time, eliminating outlying values [lo, 1 I]. We may also content ourselves with a reasonable estimate of the actual median. One way of doing this is to use a separable median filter [ 121. The image is processed by a one-dimensional horizontal median filter, the result of which is fed to a vertical median filter. The resulting image has pixel values determined as a median of median values within the filter window. The median of medians is a good statistical estimator for the actual median, with sub- stantially reduced computational complexity. Although the above techniques greatly improve the speed of median filtering, they fall far short of real- time requirements. Specialised hardware, often based upon parallel techniques, has been suggested to over- come this disadvantage. A good survey of these
Transcript
Image and Vision Computing 14 (1996) 277-283 ELSEVIER
Connectionist median filtering networks
Andrew Hunter
School of Computing and Information Systems. University of Sunderland. Priestman Building, Green Terrace, Sunderland SR2 7EE, UK
Received 7 July 1995; revised 5 September 1995
Abstract
A family of connectionist rank filtering networks is presented, suitable for real-time median filtering. It is demonstrated that these have low complexity (from 0(N2) processing elements and connections for a single output N element filter to O(N*) for a fully parailelised N x N filter). The networks have some unusual requirements in terms of processing elements and topology compared with the standard neural network models. The feasibility of VLSI implementation of the networks is discussed.
Keywords: Median filtering; Connectionist networks; VLSI implementation
1. Introduction
Median filtering is a widely used non-linear technique in image processing [1,2]. It belongs to the class of local operators, where the output value of a pixel is determined from input pixel values within a small neigh- bourhood of the output pixel (also referred to as the ‘window of the filter’). The neighbourhood is usually square, although straight and diagonal cross shapes are also used; it is invariably symmetric about a centre point. The median is calculated by ranking the input pixel values according to magnitude and selecting the middle value, i.e. for set S = {Si : 0 5 i < N},
me49 = Q((N-I)/z), where the rank R is defined by sR(i+ r) > SR(i), Y’i. As such, median filtering is an example of rank filtering [2], in which input pixels are ordered and the output pixel is set to the input pixel of a particular rank. Other important rank filters are the maximum and minimum filters, max(S) = sRCN_ ,), min(S) = sRCo).
The useful properties of median filters have been widely studied [2-61. These include the smoothing of noise and the preservation of edge shapes. Median filters are considered particularly effective for the removal of impulse noise [5,6], which is usually caused by bit errors during data capture or transmission. This combination of properties makes median filtering suitable for pre-processing of images prior to the appli- cation of other techniques for image enhancement and segmentation.
0262~8856/96/$15.00 0 1996 Elsevier Science B.V. All rights reserved SSDZ 0262-X856(95)01068-8
The most significant drawback of the median filter is its computational complexity. Ndively, the median is produced by sorting the set of input values and selecting the middle ranked value. The quickest sorting algo- rithms have cost O(N log2 N) for N elements [7]. Various authors have suggested improved techniques for the implementation of median filters. Some of these [8,9] take advantage of the fact that as the filter window is moved across the input image, only some of the pixels change. A histogram or sorted list can be maintained, with old values being removed and new ones added at each move. This method is considerably faster than conventional algorithms, especially for larger windows. Another approach is to calculate the median by sorting the pixel values one bit at a time, eliminating outlying values [lo, 1 I]. We may also content ourselves
with a reasonable estimate of the actual median. One way of doing this is to use a separable median filter [ 121. The image is processed by a one-dimensional horizontal median filter, the result of which is fed to a vertical median filter. The resulting image has pixel values determined as a median of median values within the filter window. The median of medians is a good statistical estimator for the actual median, with sub- stantially reduced computational complexity.
Although the above techniques greatly improve the speed of median filtering, they fall far short of real- time requirements. Specialised hardware, often based upon parallel techniques, has been suggested to over- come this disadvantage. A good survey of these
278 A. Hunter/Image and Vision Computing I4 (1996) 277-283
approaches is presented in Ref. [13]. The earliest system is an analogue circuit for selecting a given ranked value from several voltages [ 14,151. Other authors have suggested hardware implementations of the algorithms discussed above. The basic sorting algorithm can be used with a parallel hardware sorting circuit [ 161. The method of successive bit-wise refinement can also be imple- mented in hardware [10,13]. Another alternative is to exhaustively compare all inputs, determining the ranks of each in parallel. This is the basis of the method presented below.
This paper describes a new method for parallel median filtering based upon a simple neural network paradigm. The resulting structure is extremely regular and locally connected, making it suitable for VLSI implementation.
2. Connectionist median filter
The new technique suggested below hinges on the following simple observation. Take a set of distinct numbers S = {si; 0 < i < N}, where N is odd. The rank of an element, R(q), is equal to the number of elements smaller than it. The median is the element with rank (N - 1112.
The algorithm works as follows:
1 In parallel, compare each element in the set with every other element. Each comparator Ci,j assumes a value of 0 or 1, depending on whether element si is greater or less than element sj:
Ci,j = 0, if si > sj,
1, otherwise.
2. In parallel, sum the values of all comparators for each element to calculate the ranks:
Ri = C Ci,j. j#i
3. Extract the element with rank (N - 1)/2. This is the median:
N-l med(S) = sk, s.t. Rk = -.
2
The algorithm is implemented using a specialised The network illustrated extracts the median from a form of a neural network [19]. Neural networks consist set of N = 3 inputs. The inputs are shown on layer of a massively interconnected set of simple processing zero. These elements are assumed to fire with strength elements, or neurons. The characteristics of an individual equal to the intensity values. Layer one contains the neuron are deceptively simple. Each receives a number comparator elements. These are conventional formal of inputs from other neurons and may generate a single neurons with a step activation function. Each compara- output. The inputs are summed together, and the result tor has two inputs, the first with weight 1, the second compared with a threshold value. If the sum exceeds with weight -1. The threshold is 0. Thus, the units fire the threshold, then the neuron fires, i.e. delivers an out- if the first input is strictly greater than the second. Layer put signal to further connected neurons. The connections two contains the ranking elements. Each element has
_ Weight +1 connection
__* Weight -1 connection
0 Standard neuron
0 = Equality neuron
8 Channel switch
Fig. 1. Simple three element median network.
between the neurons each have a weight which defines how strong a signal is delivered. The major focus of research attention has been on the ability of neural networks to learn to distinguish and remember input patterns by altering the weights and thresholds of the network. However, recently researchers have discovered that non-adaptive networks with highly regular struc- tures can be used to perform fundamental operations such as the Fourier transform on images [20]. It is likely that the human brain performs similar low-level transformations using non-adaptive structures early in the vision process. It is of interest to determine for what types of filtering neuronal structures are suitable, both to design artificial networks for use in image pro- cessing, and to shed light on the functioning of the human visual system.
The network described below (see Fig. 1) is an example of a highly regular non-adaptive neuronal structure. A feature of this network is that the processing elements are not all simple formal neurons, as commonly defined. Moreover, although it is theoretically possible to repli- cate the features of this network using a network consist- ing only of formal neurons, it would be difficult and inconvenient to do so.
A. Hunter/Image and Vision Computing 14 (1996) 277~-283 279
N - 1 = 2 inputs, each with weight 1, from the com- parators. These units are specialised to test for equality with a special value (which replaces the threshold of a formal neuron) called the ‘success value’. This is set to (N - 1)/2 = 1 - the rank of the median. Only the ranking element corresponding to the median will fire. Layer three consists of specialised switching units. They have a channel and a control input. Their pattern of firing consists of allowing through the channel value if the control input is non-zero. The channel is connected to the input value in layer zero and the control to the output of the corresponding ranking element in layer two. Since only the element asso- ciated with the median receives non-zero input, only it will fire, putting the median value on the output. The fifth layer is specialised to add its inputs and put the result on the output without any thresholding. Since only one input will fire, and this will have the median value on it, the fifth layer outputs the median value.
Various improvements can be made to this scheme. First, filters for arbitrary rank can be produced by setting the success value of the layer two ranking units to zero, and connecting a single new rank setting element to each of these with connections of strength - 1. The rank setting element fires with the desired rank.
Second, we can remove half of the comparator units from layer one, since each pair of inputs is actually compared twice. Only the comparators Ci,j, where j > i, are used. If Ci,i is needed, we instead use Ci.i, weighted by a - 1 connection. The sums of the compara- tors then evaluate the rank minus i, so we add i to each
Weight +2 connection
Weight +1 connection
Weight -1 connection
Standard neuron Channel switch
Equality neuron 0 + Sumnation unit
0
1
unit
2
3
4
ranking element using a special bias element. Formally,
Ci,j = {
0, if Si > Sj
1 if Si < Sj
* Cj.i = 1 - Ci,j
R(si) = z ci.,j = C ci,j + C ci,,
i#i j<r j>r
= CC1 - cj.i) + Ccl.,
j<i j>i
=i+-1 CCj,i +Cct,j f 1
See Fig. 2.
\i<r / j>i
Removing redundant comparisons in this fashion also allows the network to handle non-unique (repeated) input values. Consider a five-input version of the original network with input set {0,2,0,1, O}. The median is clearly 0, since the element of rank 2 must have this value. The simple network will generate a set of corres- ponding ranks {0,4,0,3,0} by counting how many other elements each is strictly greater than. No element of rank 2 is present, causing the network to fail to find a median. However, the improved network calculates the set of ranks (0,4,1,3,2}, thus allowing a valid median (or any other rank) to be found. This property occurs because the modified network makes a consistent decision about the relative sizes of pairs of equal valued inputs, by using the array position to disambiguate. Specifically, if two values are equal then the rightmost one is considered larger.
3. Multiple output one-dimensional filter
The network described in the previous section can be improved by using further parallelism. Rather than determining the value of one output, we can calculate many at a time. Many of the level one comparator
Fig. 2. Improved three element median network. Fig. 3. Part of a multiple three element median network
280 A. Hunter/Image and Vision Computing 14 (1996) 277-283
units can be shared by adjacent ranking units, effect- ing a significant reduction in cost (from O(N2) to O(N)). Fig. 3 shows a section from a multiple output one-dimensional three element rank filter; bias and rank setting units have been omitted for clarity. This network implements a one-dimensional filter across a row of intensity values, whereas two-dimensional filtering is the norm in real image-processing; it is included principally to introduce important concepts reused in the two-dimensional case, although it could also be used directly in implementing a separable median filter. (4
The complexity of the network can be described in terms of the number of processing units and number of connections. Discounting edge effects, the cost of an N element rank filter of width W is summarised layer by layer in Table 1. The total cost is 3 WN + W processing units and WN2 + 4WN - 2 W connections; that is, 0( WN) processing units and 0( WN2) connections. The most explosive part of the cost is clearly contributed by the connections between layers one and two (com- parators to ranking units). It is possible to reduce this number of connections to 0( WN) at the cost of some more simple processing units (specifically, two-input summation units). This technique only offers reductions in numbers of connections for N > 4, and it is note- worthy that connections are cheaper to build than processing units. However, for significant values of N it may prove worthwhile.
2
(b)
Fig. 4. Reducing connections in a five element network. (a) An extract from the standard network, (b) simplified extract using extra summa- tion units.
The technique is best described with the aid of a diagram (see Fig. 4). We exploit the fact that common sets of comparators are used by multiple ranking units. We combine the output of such sets of comparators using a summation element, then feed its output to the ranking units. The pattern is not easily discerned in the normal layout, so the order of the processing units has been changed in Fig. 4. The comparators have been arranged symmetrically about one taking adjacent inputs, in such a fashion that a set of p comparators, including the middle one and extending either leftwards or rightwards, corresponds to p + 1 contiguous inputs. These sets, in turn, are shared by N -p ranking units. The alternative structure, shown in Fig. 4(b), builds up such sets of various sizes, which are then fed to the
ranking units. There are W such sub-networks in the entire network, accounting for all the layer one to layer two connections.
The sub-network in Fig. 4(a) contains N(N - 1) con- nections from layer one to layer two. The replacement in Fig. 4(b) contains 6(N - 2) connections and 2(N - 3) summation units (the connections are counted as N - 2 on each diagonal, plus N - 2 above and N - 2 below). Thus, it is possible to define a rank filtering network which is 0( WN) in both numbers of processing units and numbers of connections. If a ratio is available for the relative costs of processing units and connections, these figures can be used to determine which approach is more efficient for a specific value of N.
4. Two-dimensional filtering
Table 1 Network cost for multiple-output 1D filter
Processing elements Connections
Layer Elements Layers
0 W o-1 1 W(N-1) l-2 2 WN 2-3 3 WN 3-4 4 W o-3
Connections
2w (N- 1) Wiv(N-1) WN WN WN
The previous sections have described how to con- struct a one-dimensional connectionist rank filter. Real image processing applications use two-dimensional filtering. Since it becomes near impossible to draw legible diagrams of the two-dimensional case, this section is principally narrative, and draws upon results established in the one-dimensional case.
We will consider two forms of two-dimensional filter. The first is a strip filter, which produces as output a single row of W values. The input is N x W. The strip
A. Hunter/Image and Vision Computing 14 (1996) 277-283 281
filter can be used to process an entire image one row at a time. The second is a multi-row filter, which produces H rows simultaneously. A full implementation would process an entire image in one application of the net- work; a less ambitious approach would process standard depth strips of the image.
The strip filter effectively passes an N x N window across the strip to calculate the W output values. The associated costs are summarized in Table 2. The number of ranking units is calculated as N* for each of the W outputs. Each ranking unit draws on N* - 1 com- parators, so there are WN2(N2 - 1) connections from layer 1 to layer 2. To determine the number of com- parators in layer 1, consider moving the filter window horizontally across the strip. If two inputs are p columns apart, N -p windows will contain both. Since a com- parator is used twice in each application of a filter window, each is used 2(N -p) times. The average number of uses is given by:
&N-p)) =N+l
N
The number of comparators is therefore WN2(N2 - l)/ (N + 1) = WN*(N - 1).
It can be seen that the most explosive cost is again in the connections from layer 1 to 2 (comparators to rank- ing units), which is 0( WN4). This cost can be reduced by introducing a new layer of column-sum elements between the comparators and ranking elements. A column-sum is formed by summing the comparators of a given element with each of the units in a given column. Every ranking unit associated with the given element can then use a combination of its column-sums. The column-sum for an element’s own column is formed from N - 1 com- parators; those for other columns are formed from N comparators. Each ranking element draws input from precisely N column-sum units, so there are a total of WN3 inputs to layer 2. Each input element is associated with 2N - 1 column-sums, so there are WN(2N - 1) new column-sum units (which are N-input or (N - 1)-input summation units). Each set of 2N - 1 column-sums is formed using N(2N - 2)+
Table 3 Network cost for N x N 2D multi-row filter
Processing elements Connections
Layer Elements Layers Connections
0 WH o-1 4WHN (N-l)
1 2WHN(N-1) l-2 WHN’ (N2 - 1) 2 WHN* 2-3 WHN2
3 WHN* 3--4 WHN*
4 WH o-3 WHN*
(N- 1) = (2N+ l)(N- 1) comparators, so there are WN(2N + l)(N - 1) connections between comparators and column-sum units. In total, there are 0( WN3) connections (both from and to the column-sum units) and 0( WN*) extra summation units.
A further reduction in the number of connections can be effected on the connections between column- sum units and ranking units, using a method similar to that illustrated in Fig. 4. Column-sums to the right are accumulated, as are those to the left, and the ranking units take three inputs: a left sum, a right sum and the same column column-sum. This structure costs 6(N - 2) + N connections and 2(N - 3) extra two-element summation units (O(WN*) across the whole filter). However, the number of comparator to column-sum connections remains 0( WN3), so the total cost of the network remains 0( WN*) units and 0( WN 3, connections.
The situation is different again if we consider a filter which handles multiple rows. Table 3 summarises element and connection costs (omitting edge effects) for an H rows filter. All costs are 0( WHN*) or less, apart from the familiar layer one to two connections, which are 0( WHN4). We can use column-sum units to reduce this cost as before. Each of the WH input units has associated with it N(2N - 1) column-sums (account- ing for the N vertical shifts), so there is a total of WHN(2N - 1) column-sum units, which is 0( WHN2). As in the strip filter, the ranking units have precisely N inputs each, so there are WHN3 connections between the column-sum units and the ranking units, which can be reduced to 0( WHN2) using the technique described above for the strip filter. The total number of connec- tions between comparators and column-sum units is WHN(2N + l)(N - l)), which is 0( WHN3). Using the reduction technique from Fig. 4 again, this time applied vertically to the formation of column-sums for each element, we can reduce the number of connections to 0( WHN*), at the cost of 0( WHN*) more two-element summation units. The total cost of the network is then 0( WHN*) units and 0( WHN*) connections - that is, directly proportional to the number of pixels filtered times the size of the filter window. Intuitively, this would seem to be as good an efficiency rating as we could expect.
282 A. Hunter/Image and Vision Computing 14 (1996) 277-283
5. VLSI implementation of the rank-filtering networks
We have already established that the complexity of the networks (in terms of processing elements and con- nections) is reasonable. If we also recall that the struc- tures are extremely regular and repetitive, locally connected, and that the individual processing elements are fairly simple, then we may conclude that the net- works are suitable for VLSI implementation. However, we must also establish whether the sheer scale of the networks makes VLSI implementation feasible.
Without delving into specific VLSI implementation strategies, we can make some order-of-magnitude esti- mates of the complexity. Modern chips can supports tens of millions of transistor devices; given the com- plexity of the units described here (and assuming eight bit grey-scale values), a single chip might support of the order of 10,000 processing units.
If we assume an image width of 1024 pixels, and a 3 x 3 filter (the most commonly used), we can produce the following approximate requirements (in terms of number of units):
This ignores the silicon area cost of connections, which must also be taken into account (there are trade-offs available, given that the networks may include additional units to reduce numbers of connections). Nevertheless, we may draw some tentative conclusions:
l The 1D filter could be implemented in current tech- nology; if it has both horizontal and vertical capabili- ties, then a separable medium filter is achievable.
l The strip filter is probably achievable if a relatively small strip (perhaps 128 or 256 pixels) is handled.
l The full two-dimensional filter is of theoretical interest only (unless a small area filter, perhaps up to 16 x 16 pixels, is constructed).
In all cases, repeated application of a small filter can achieve the effects of a larger filter on a slower time-scale. Real-time image-processing requires filtering on the order of 50 frames/s. Given modern processing capabilities of tens of millions of instructions per second, this allows hundreds of thousands of cycles per filter. At this speed, even quite a small filter such as the 128 pixel strip-filter mentioned above could process an entire image on the required time-scale.
6. Limitations of the networks
The networks described here suffer from a number of limitations. First, they are unable to filter using differ- ent sized windows. The single output filter could be
used for different size filters, by padding the input with minimum values and adjusting the rank selected accord- ingly; however, the multi-output filters would have to have extra logic added between input and comparator layers. Second, there may be problems with edge effects, if the strip filter or full two-dimensional filter are to be used for images of varying size. One solution is to build a network with W + N inputs but only W outputs. The picture is pre-processed, padding the extra N/2 edge pixels on each side with values obtained by an appro- priate technique (e.g. mirroring about the image edge). Similarly, the full two-dimensional filter would be H + N inputs high. Simple specialised hardware could perform the padding operation prior to activat- ing the network. On the other hand, if the size of images is constant the network hard-codes the image edge handling method by appropriate alterations to the connections at the edges. A network hard-coded in this fashion can still be used for smaller images if they are appropriately padded, or even for larger images with appropriate cutting and pasting of processed sub-images, so the choice of approach is based on an assessment of the most regular mode of use, rather than being absolute.
7. Conclusion
This paper has presented a family of rank filtering connectionist networks. The networks are well-suited to specialised VLSI implementation, being highly regular in structure, locally connected and having acceptable costs (as low as 0( WHN*) for an N x N parallel filter across a W x H image). These networks demonstrate how some simple extensions to the com- monly accepted neural network paradigm makes it easy to perform operations which would otherwise be extremely difficult. This is indicative of the need to con- sider wider forms of processing than the threshold firing neuron in developing connectionist technology for specialised applications.
References
[l] T.S. Huang (ed.), Two-dimensional,Digital Signal Processing II: Transforms and median filters, Springer-Verlag, New York, 198 1.
[2] R.M. Hodgson, D.G. Bailey, M.J. Ng and S.J. McNeill, Proper- ties, implementations and applications of rank filters, Image and Vision Computing, 3 (1) (1985) 3-14.
[3] A.C. Bovik, T.S. Huang and D.C. Munson Jr., The effect of median filtering on edge estimation and detection, IEEE Trans. Pattern Anal. Machine Intell., 9 (2) (1987) 181-119.
[4] N.C. Gallagher Jr. and G.L. Wise, A theoretical analysis of the properties of median filters, IEEE Trans. Acoust. Speech, Signal Process., 29 (1981) 1136-1141.
[5] T.A. Nodes and N.C. Gallagher Jr., Median filters: some modifi- cations and their properties, IEEE Trans. Acoust. Speech, Signal Process., 30 (1982) 739-746.
A. Hunter/Image and Vision Computing 14 (1996) 277-283 283
[6] E. Ataman, V.K. Aatre and K.M. Wong, Some statistical proper-
ties of median filters, IEEE Trans. Acoust., Speech, Signal
Process., 28(4) (1980) 415-421.
[7] D. Knuth, The Art of Computer Programming Vol. 3, Sorting
and searching, Addison-Wesley, Reading, MA, 1975.
[8] T.S. Huang, G.J. Yang and G.J. Tang, A fast two-dimensional
median filtering algorithm, IEEE Trans. Acoust. Speech, Signal
Process., 27 (1) (1979) 13-16.
[9] J.B. Bednar and T.L. Watt, Alpha trimmed means and their
relationship to median filters, IEEE Trans. Acoust., Speech,
Signal Process., 32 (1984) 145-153.
[lo] E. Ataman, V.K. Aatre and K.M. Wong, A fast method for
real-time median filtering, IEEE Trans. Acoust., Speech, Signal
Process., 28 (1980) 415-420.
[ll] P.E. Danielsson, Getting the median faster, Comput. Graph.
Image Process., 17 (1) (1981).
[12] P.M. Narendra, A separable median filter for image noise
