Similarity and Clustering of Footwear Prints

Yi Tang, Sargur N. Srihari and Harish KasiviswanathanDepartment of Computer Science and Engineering

Center of Excellence for Document Analysis and Recognition (CEDAR)University at Buffalo, The State University of New York

Amherst, New York 14228, U.S.A{yitang, srihari, harishka}@buffalo.edu

Abstract—Research on footwear impression evidence hasbeen gaining increasing importance in forensic science. Givena footwear impression at a crime scene, a key task is tofind the closest match in a local/national database so as todetermine footwear brand and model. This process is madefaster if database prints are grouped into clusters of similarpatterns. We describe a clustering approach based on commonprimitive patterns. Shape features consisting of lines, circlesand ellipses are extracted from database prints using variationsof the Hough transform. Then an attributed relational graph(ARG) is constructed for each known print, where each node isa primitive feature and each edge represents a spatial relation-ship between nodes. A footwear print distance (FPD) betweenARGs is used as similarity measure. The FPD is computedbetween each known print and pre-determined patterns to formclusters. The use of the methodology is demonstrated with alarge database of known prints.

Keywords-footwear evidence; similarity; clustering; Houghtransform; content-based image retrieval

I. INTRODUCTION

Various types of impression evidence, such as finger-prints and footwear prints are commonly present in crimescenes. Footwear impressions, which are the most commonlypresent evidence, have been relatively less used in foren-sic analysis [1]. This is because footwear impressions areusually highly degraded, prints are inherently complex anddatabases are too large for manual comparison. Their use isrecently gaining importance with the potential of computer-assisted methods.

Most existing footwear print retrieval systems are semi-automatic. De Chazal et al. [2] proposed a fully automatedshoe print classification system which uses power spectraldensity of the print as a pattern descriptor; crucial infor-mation of the print is preserved by removing low and highfrequency components. Zhang et al. [3] proposed an auto-mated shoe print retrieval system in which edge directionhistogram is used to find the closest matching print. Thereis no published literature on mining footwear print databasesto aid in retrieval. As an exercise in data mining, Sun et. al.[4] clustered shoe outsoles using color (RGB) informationas features where the number of clusters k was varied from2 to 7 and the clustering results of k-means and expectation

(a) (b) (c) (d) (e) (f)

Figure 1: Sample footwear prints in database showing presenceof primitive shapes.

maximization were compared; the results are of limited usesince RGB information of outsole photographs are absent inimpression evidence.

Retrieving the most similar prints to an impression canbe made faster by clustering the database prints beforehand.Outsoles of footwear have common primitive patterns likecircles, ellipses, triangles and wavy patterns [5], [6] whichwe make use of to cluster footwear prints. Sample printsin a database are shown in Figure 1. Primitive shapes oflines, circles and ellipses are used as features from whicha description of the footwear print is constructed as anAttributed Relational Graph (ARG) [7], [8]. The similaritybetween ARGs is computed using a footwear print distance(FPD).

A similarity measure being essential for clustering, weuse a variation of the earth mover distance (EMD) betweenimages, a technique used in content based image retrieval[9]. EMD has been used in in shape matching [10] andcommon pattern discovery [11]. It has been used in speakerclustering [12]; where it is applied to the MIXMAX speakermodel which improves processing speed of hierarchicalclustering.

II. FEATURE EXTRACTION

The first step of retrieval is feature extraction. Color,texture and shape can be used to distinguish images [13].We chose shape features as the basis of distance/similaritycomputation. Color features are absent as the acquired printsmade under a controlled environment are gray-scale images.Textures are sensitive to acquisition methods and susceptible

Figure 2: Process flow for clustering footwear prints in database.

to wear while shapes are resistant to wear and present overa long period of time. Shape features are also robust againstocclusion and incompleteness, i.e. the wear or variation of alocal region on the outsole will be less likely to affect shapefeatures in other regions.

In the clustering approach proposed (Figure 2) three basicshapes are detected: straight line segments, circles/arcs andellipses. This is based on the analysis of 3,000 footwearprints that 10% of the prints have ellipse(s), 30% of themhave circles/circular arcs and almost every footwear printhas line segments which are at least 25 pixels long. Anyshapes other than circles and ellipses are approximated bypiecewise lines. Combination of these shapes can be used touniquely identify the pattern of a footwear print.

The Hough transform [14] detects structures satisfying aparametric form in an image by mapping foreground pixelsinto parameter space characterized by an n-dimensionalaccumulator array, where n depends on the number ofparameters used to describe the shape of interest in Houghspace. Each significant pixel from the shape of interestwould cast a vote in the same cell of an accumulator array,hence all pixels of a shape gets accumulated in a singlecell. Number of valid peaks in the accumulator array wouldcorrespond to the number of shapes in the image.

Line Detection: Normal representation of a straight linetakes two parameter and hough transform maps each pixelin the straight line to an 2-D accumulator array.

Circle Detection: It involves building a 3-dimensionaccumulator array. Gradient orientation [15] is used to limitthe generation of spurious votes. Further, spatial constraintsare used to eliminate spurious circles.

Ellipse Detection: In a Cartesian plane, an ellipse canbe described by its centre (p, q), length of the semi-majoraxis a, length of the semi-minor axis b and the angle θbetween the major axis and the x-axis. Hence the five pa-rameters (p, q, a, b, θ) [16] are required to uniquely describe

an ellipse. These five parameters demand a five dimensionalaccumulator which is computationally expensive but Ran-domized Hough transform (RHT) [17] for ellipse detectionis computationally advantageous. In case of ellipse detectionin footwear prints, RHT cannot be used directly. This isbecause there are around 50,000 foreground pixels in afootwear print of typical size 600 × 800 and picking threeforeground pixels from them in random will never narrowdown to the right ellipse. Therefore connected componentanalysis was first performed on the edge image of thefootwear prints. Then each of the connected component wasscanned for an ellipse using RHT. To reduce the computationcertain connected components were eliminated based onthe eccentricity property of the region enclosed by eachcomponent.

Parameters of the ellipse are determined analogous toMcLaughlin’s approach [17]. The algorithm to find theparameters of the ellipse consists of the following steps.

Step 1: Pick three foreground pixels p1, p2 and p3

randomly and fit a tangent at each of the picked point,namely t1, t2 and t3.

Step 2: Find the intersection of the tangents t1 and t2,and intersection of tangents t2 and t3.

Step 3: Find the straight line that passes through themidpoint of pixels p1 and p2 and the intesection of theirtangents. Repeat the same step with pixels p2 and p3. Theintersection of the two lines gives the centre of the ellipse.

Step 4: Shift the ellipse centre to the origin to get rid ofthe parameters D and E in the conic equation.

Ax2 +Bxy + Cy2 +Dx+ Ey + F = 0 (1)

Step5: Find the coefficients A, B and C in conic equationby substituting the co-ordinates of the three picked pointsand by solving the system of linear equations.

Validation of Detected Ellipses: In a highly degradedfootwear print spurious ellipses are detected. To overcome

(a) (b)

Figure 3: Eliminatioin of spurious ellipses using gradient orienta-tion. (a) detected ellipses before validation, and (b) detected ellipsesafter eliminating spurious ones.

this we check for the follwing conditions:1. Find the foreground pixels that satisfiy the equation

(2) of the detected ellipse. These pixels are the potentialcandidates to be a true ellipse pixel.

((x−p) cos θ+(y−q) sin θ)2

a2 + ((y−q) cos θ+(x−p) sin θ)2

b2 = 1 (2)

2. At each candidate pixel, check for an angle of 90 degreebetween their gradient direction and gradient orientation.Pixels that satisy this condition is a true ellipse pixel.The fraction of the true ellipse pixels to the perimeter ofthe ellipse gives the measure of the quality of the ellipse.Detected ellipses before and after validation are shown inFigure 3. Results of Feature extraction are shown in Figure4.

III. ATTRIBUTED RELATIONAL GRAPH (ARG)

Relational structures have been used to represent complexobjects and scenes [18] for image matching. Graph represen-tation [19] has great advantage over feature vector because ofits ability to explicitly model relationship between differentparts and feature points.

After feature extraction, a footwear print has been de-composed into a set of primitives. To obtain a structuralrepresentation of these primitives, an Attributed RelationalGraph(ARG) [7], [8] is built for each print. An ARG is a3-tuple (V; E; A) where V is the set of nodes, E is theset of edges and A is the set of attributes. Lines, circlesand ellipses are defined as nodes. Each edge describes thespatial relationship between nodes. The attributes includenode attributes (unary) and edge attributes (binary).

There are three types of nodes (lines, circles and ellipses),and nine types of edges (line-to-line, line-to-circle, line-to-ellipse, circle-to-line, circle-to-circle, circle-to-ellipse,ellipse-to-line, ellipse-to-circle and ellipse-to-ellipse denotedas L2L, L2C, L2E, C2L, C2C, C2E, E2L, E2C, E2E edges).To tackle the case of nodes being missing or incorrectly

(a) (b) (c)

(d) (e) (f)

Figure 4: Results of Feature Extraction. Detected lines, circles andellipses are shown in green, red and blue respectively.

detected due to noise, occlusion and incompleteness, a fully-connected directed graph is adopted. This means that thereis a directed edge from each node to all other nodes. Todistinguish one node from the other or one pair of nodesfrom another, node and edge attributes have been carefullydefined to quantify the spatial relationship between eachpair of nodes in terms of distance, relative position, relativedimension and orientation. All these attributes have beennormalized in the range [0, 1]. This description is invariantto scale, rotation, translation and insensitive to noise anddegradations. It is also similar to the way humans distinguishone set of shapes from the other.

IV. FOOTWEAR PRINT DISTANCE (FPD)

The quality of clusters depends highly on the similaritymeasure used by the clustering algorithm so the similaritymeasure has to be accurate and robust. Common similar-ity measures are: Euclidean distance, Manhattan distance,correlation similarity etc. A distance metric, known as theWasserstein metric, or the Earth Mover’s Distance (EMD)between probabiity distributions is popular in content-basedimage retrieval [9]. The EMD evaluates the least amountof work that is needed to transform one distribution into theother. EMD is a true metric if the ground distance is a metricand the total weight of the two signatures are equal.

We propose Footwear Print Distance (FPD), which isderived from EMD, to compute the similarity betweenfootwear prints. Let SP = (V,E,A,W,N) be a footwearprint, where V = {Vi}Ni=1 is the set of N nodes cor-responding to primitive shapes; E = V × V is the setof edges between nodes; A is the set of node and edgeattributes with A = AN ∪ AE , AN =

⋃Ni=1ANi, AE =⋃N

i=1

⋃Nj=1AEij ; and W = {Wi}Ni=1 is a set of weights.

Let SP1 = (V1, E1, A1,W1, N1) be the first footwear printwith N1 nodes; SP2 = (V2, E2, A2,W2, N2) the secondfootwear print with N2 nodes; and C = [cij ] be the unitcost matrix where cij is the unit matching cost betweenV1i ∈ V1 and V2j ∈ V2. Each node is assigned equal weightto enforce a one-to-one correspondence between nodes, i.e.W1i = W2j = 1

max(N1,N2), 1 ≤ i ≤ N1, 1 ≤ j ≤ N2.

The goal is to calculate a node correspondence matrixM = [mij ], where mij denotes the amount of weight trans-ferred from V1i to V2j , which minimizes the total matchingcost Cost(SP1, SP2,M) =

∑N1i=1

∑N2j=1mij ∗ cij , subject

to the following constraints:

mij ≥ 0, 1 ≤ i ≤ N1, 1 ≤ j ≤ N2 (3)

N2∑j=1

mij ≤W1i, 1 ≤ i ≤ N1 (4)

N1∑i=1

mij ≤W2j , 1 ≤ j ≤ N2 (5)

N1∑i=1

N2∑j=1

mij = min(N1∑i=1

W1i,

N2∑j=1

W2j) (6)

Once the correspondence matrix is found, the FPD isdefined as the overall matching cost normalized by the sumof all the weights transferred from SP1 to SP2.

FPD(SP1, SP2) =

∑N1i=1

∑N2j=1mij ∗ cij∑N1

i=1

∑N2j=1mij

(7)

For a given pair of nodes in two footwear prints, howone node is different from the other depends not only onthe nodes themselves, but also on how each of them relatesto its neighbors in terms of distance, orientation, position etc.Hence, we define the unit matching cost between these twonodes as an EMD which takes into account both the nodesand their neighbors. Dummy nodes are added in the footwearprint with less number of nodes to have equal number ofnodes in both the prints.

V. CLUSTERING USING RECURRING PATTERNS

Clustering algorithms can be generally divided intopartition-based, density-based and hierarchical based meth-ods [20]. Algorithms like K-means, Hierarchical Clustering,and Expectation Maximization requires similarity matrixconsisting of pair-wise distance between every footwear

prints in dataset. Building similarity matrix is computa-tionally expensive for a large dataset. Further, the ARGrepresenting a footwear print has 200-300 nodes on averageand nodes can vary considerably in terms of relative size,position etc. This makes the feature space very sparse andtherefore similar footwear prints tend to stay close to eachother and dissimilar ones stay apart. Hence, to cluster theentire dataset we propose using recurring patterns as fixedcluster center.

Recurring patterns [21], [22], [23] such as wavy pattern,concentric circles etc. are common in footwear prints andeach of them can be used to represent a group of similarfootwear prints. These patterns are simple in structure andgraphs constructed from these patterns have much lessnumber of nodes. Hence, recurring patterns can be usedas query to fetch all similar prints from the database. Thisdrastically reduced the computation and does not requiresimilarity matrix. Though this clustering method requiresdomain knowledge to determine the recurring patterns, itavoids the problems of deciding the number of clustersbeforehand (unlike K-means).

From visual inspection of 1000 prints, 20 recurring pat-terns(shown in Fig.5) were determined and used as clusterrepresentatives. For each database footwear print, we com-puted its FPD to each pattern and then assigned it to thenearest cluster representative. These cluster representativesare similar to cluster means in K-means algorithm but these”means” are fixed. This efficiency is achieved by exploitingthe sparseness of the feature space.

Figure 5: 20 cluster representatives

VI. EXPERIMENTS AND RESULTS

Our dataset consists of 1000 prints and each of themhave meta-data information such as brand and model of thefootwear. Most prints have a resolution of either 150 or 72DPI.

Step 1: The first step in the feature extraction is to performmorphological operations such as dilation and erosion. Thismakes the interior region of the boundary uniform and hencethe Canny edge detector [14] does not detect any edgesinside the boundary. This helps to enhance the quality ofthe edge image. Result of step 1 for a sample footwear printis shown in Figure 6.

(a) (b) (c) (d)

Figure 6: Results of Step 1. (a) Original Gray-scale Image, (b)Edge Image of (a), (c) Result of Morphological Operation on (a),(d)Edge Image of (c).

(a) (b) (c) (d)

Figure 7: Results of Step 2: Extracting features in the sequencecircle→ellipse→line. (a) Circles. (b) Ellipses. (c) Line Segments.(d) All features. Red box indicates a small region in the footwearprint

Step 2: SHT is used to detect circles in footwear prints.Pixels of detected circles shown in Fig. 7(a) are removedfrom the edge image and fed as input for ellipse detectionusing RHT. Pixels of detected ellipses shown in Fig. 7(b)are removed from the edge image and the output is fed asinput for line detection in Fig. 7(c). Features are extracted inthe order: circle, ellipse and line. This is because circles aredegenerated ellipses and arbitrary shapes in footwear printare approximated by piecewise lines. Fig. 7(d) sums up allthe features.

Step 3: For each detected feature, node attributes likecompleteness & quality of circle, eccentricity of the ellipseetc. are computed. Further, edge attributes like relativedistance & position between nodes are calculated and finallyan ARG is constructed. One such ARG is shown in Fig. 8.

Step 4: FPD between each database print and every clusterrepresentative is calculated. Then each print is assignedto the nearest representative, for which the FPD is belowthreshold T . If FPD between a print and cluster representa-tives are greater than T , then the print remains as a singlecluster. In our experiments, T was set to 0.15.

550 footwear prints were associated with one of the 20clusters wheras the remaining 450 prints were so unique

(a) (b)

Figure 8: (a) ARG for footwear print in Figure 6. (b) Subgraphfor the region enclosed in the red box of Figure 7(d). Red andGreen dots represent circles and lines respectively.

that each of them was a cluster by itself. F -measure,the weighted harmonic mean of precision and recall, iscomputed as a measure of clustering’s accuracy. PrecisionVs. Recall curve and F -measure are shown in Fig. 9(a).

F =2× Precision×RecallPrecision+Recall

(8)

(a) Precision Vs. Recall

(b) Two sample clusters

Figure 9: Clustering using recurring patterns

One advantage of this clustering method is huge re-duction in computation. For instance, for a database of1000 prints, existing clustering algorithms would require1000C2 = 499, 500 FPD computations to build the similaritymatrix. However, our clustering method would take 1000×kFPD computations, where k is the number of recurringpatterns. In our case, k = 20, so the computation is reduced

by 96%. This efficiency is achieved without compromisingthe accuracy or recall rate (shown in Fig. 9(a)). Two sampleclusters from the clustered database are shown in Fig. 9(b).

VII. CONCLUSION

In this paper, we have proposed a new approach to find thesimilarity between footwear prints and cluster them based onoutsole patterns. It is evident from the result that geometricshapes are one of the most reliable features, Attributedrelational graph is a robust descriptor of the primitivefeatures and FPD is a good similarity measure. Clusteringusing recurring patterns achieves good performance in termsof both accuracy and efficiency. Future direction is to usethe clustered database to match the crime scene mark withknown prints.

ACKNOWLEDGMENT

This work was supported by the Department of Justicegrant 2007-DN-BX-K135. The opinions expressed are thoseof the authors and not of the DOJ. We would like to thankFoster and Freeman, Ltd. for providing us datasets.

REFERENCES

[1] W. J. Bodziak, Footwear Impression Evidence: Detection,Recovery and Examination, 2nd ed. CRC Press, 2000.

[2] P. D. De Chazal and R. Reilly, “Automated processing ofshoeprint images based on the fourier transform for use inforensic science,” IEEE Transaction on Pattern Analysis andMachine Intelligence, vol. 27, pp. 341–350, 2005.

[3] L. Zhang and N. Allinson, “Automatic shoeprint retrievalsystem for use in forensic investigations,” 5th Annual UKWorkshop on Computational Intelligence, 2005.

[4] W. Sun, D. Taniar, and T. Torabi, “Image mining: A case forclustering shoe prints,” International Journal of InformationTechnology and Web Engineering, vol. 3, pp. 70–84, 2008.

[5] S. Mikkonen and T. Astikainen, “Databased classification sys-tem for shoe sole patterns - identification of partial footwearimpression found at a scene of crime,” Journal of ForensicScience, vol. 39, pp. 1227–1236, 1994.

[6] G. Alexandre, “Computerized classification of the shoeprintsof burglars soles,” Forensic Science International, vol. 82, pp.59–65, 1996.

[7] A. Sanfeliu and K. S. Fu, “A distance measure betweenattributed relational graphs for pattern recognition,” IEEEtransactions on systems, man, and cybernetics, vol. 13, pp.353–362, 1983.

[8] H. Bunke and B. T. Messmer, “Efficient attributed graphmatching and its application to image analysis,” in Proceed-ings of the 8th International Conference on Image Analysisand Processing, 1995, pp. 45–55.

[9] Y. Rubner, C. Tomasi, and L. J. Guibas, “The earth moversdistance as a metric for image retrieval,” International Journalof Computer Vision, vol. 40, pp. 99–121, 2000.

[10] K. Grauman and T. Darrell, “Fast contour matching using ap-proximate earth movers distance,” in CVPR ’04: Proceedingsof International Conference on Computer Vision and PatternRecognition, 2004, pp. 220–227.

[11] H.-K. Tan and C.-W. Ngo, “Common pattern discovery usingearth mover”s distance and local flow maximization,” in ICCV’05: Proceedings of the Tenth IEEE International Conferenceon Computer Vision. Washington, DC, USA: IEEE ComputerSociety, 2005, pp. 1222–1229.

[12] T. Stadelmann and B. Freisleben, “Fast and robust speakerclustering using the earth mover’s distance and mixmaxmodels,” in Proceedings of the IEEE International Conferenceon Acoustics, Speech and Signal Processing, 2006, pp. 989–992.

[13] Y. Rui, S. Huang, and S. Chang, “Image retrieval: Currenttechniques, promising directions, and open issues,” Journalof Visual Communication and Image Representation, vol. 10,pp. 39–62, 1999.

[14] M. Nixon and A. Aguado, Feature Extraction and ImageProcessing. Elsevier Science, 2002.

[15] J. Goulermas and P. Liatsis, “Incorporating gradient esti-mations in a circle-finding probabilistic hough transform,”Pattern Analysis and Applications, vol. 26, pp. 239–250,1999.

[16] W. Y. Wu and M. J. J. Wang, “Elliptical object detection byusing its geometric properties,” Pattern Recognition, vol. 26,pp. 1499–1509, 1993.

[17] R. McLaughlin, “Randomized hough transform: better ellipsedetection,” IEEE TENCON-Digital Signal Processing Appli-cations, vol. 1, pp. 409–414, 1996.

[18] R. M. Haralick and L. G. Shapiro, Computer and RobotVision. Addison Wesley, 1992.

[19] H. Bunke, C. Irniger, and M. Neuhaus, “Graph matchingchallenges and potential solutions,” International Conferenceon Image Analysis and Processing, vol. 3617, pp. 1–10, 2008.

[20] M. Aldenderfer and R. Blashfield, Cluster analysis. SAGE,1984.

[21] A. Girod, “Computerized classification of the shoeprint ofburglars shoes,” Forensic Science Int., vol. 1, pp. 59–65, 1982.

[22] Foster and Freeman, “Solemate.” [Online]. Available:http://www.fosterfreeman.com/

[23] S. Mikkonen, V. Suominen, and P. Heinonen, “Use offootwear impressions in crime scene investigations assisted bycomputerised footwear collection system,” Forensic ScienceInt., vol. 82, pp. 67–79, 1996.