JOHN Y. C. WANGU. S. Army Engineer Topographic Laboratories

Fort Belvoir, VA 22060

Feature Extraction byInteractive Image Processi ng

The computer aids the operator in detecting and delineatingcultural features on digitized imagery.

INTRODUCTION

A N IMPORTANT SUBJECT before scien­tists, engineers, and cartographers at

the present time is the rapid, accurate, andeconomical processing of imagery that de­picts the surface of the earth or other planetsremotely sensed from space vehicles. Inmapping we are interested in planimetric

the original picture into simpler form, e.g., aline drawing, by some sequence of steps.Each step should preserve the importantproperties of the original, while reducingthe computational requirements imposed onsubsequent steps. But the reduced picturegenerally contains less information than the

ABSTRACT: In aerial photography, images of roads and other man­made structures are small compared to those of natural features.Their edges are usually indistinct from the background and often goundetected by automatic systems. A major problem encountered byautomatic detection systems is the delineation ofboundaries aroundareas containing features of interest. The computer is often unableto accomplish the detection process alone. Therefore, it is reasonableto combine the superior pattern recognition abilities of a humanbeing with the computational power of a digital computer to forman interactive system. Such an interactive feature extraction systemis discussed. The objective is to generate the photo coordinates forfeature boundaries from aerial photographs. The hardware consistsof a CDC-6400 sequential computer, a Goodyear STARAN associa­tive array processor, a DEC PDP 11/50 minicomputer, and an imagedisplay, as well as digitizing and recording devices. The featureextraction program for the interactive system consists of a series ofroutines which were designed, written, and installed in the systemby the author. The routines were written in FORTRAN and/or theSTARAN assembly language according to the intended applicationin order to use the computers efficiently.

features such as roads, built-up areas, drain­ages, and natural boundaries; however, trac­ing planimetric features from aerial photo­graphs rapidly, accurately, and economicallyis not a simple task. Presently, automatedcartographic instruments can produce a vari­ety of maps, but cannot compile line mapscompletely automatically. Cartographic de­tails still must be extracted manually byhuman operators. The extraction of featuresis the bottleneck in the total automation ofthe mapping processes.

Image processing methods may convert

original, and there is no guarantee that theinformation lost is irrelevant. A number ofmethods to determine planimetric featuresautomatically from digital data may serve forsome types of imagery or for some specialpurpose3.4, however, there is no single stan­dard approach to the extraction of featuresfrom remotely sensed data. In practice, it isnot always wise to attempt to map objectsinto clear-cut, two-value overlays. Very oftenobjects are defined only in a "fuzzy" senseand no definite or distinct edge can bedrawn by automatic methods.

PHOTOGRAMMETRIC ENGINEERING AND REMOTE SENSING,

Vol. 43, No. 12, December 1977, pp. 1495-1501.

1495

1496 PHOTOGRAM METRIC ENGINEERING & REMOTE SENSING, 1977

In aerial photography, roads and build­ings are very small compared to fields andforests, and their edges usually are not dis­tinct from the background. In small-scalephotography, roads are usually imaged as awhite line with a small surface area. Thesemay often go undetected by automatic sys­tems while features of less interest to mapmakers, such as fields, are easily detected.Even with a good edge extraction method,there remains the problem of decidingwhich edges should be extracted to give themaximum amount of information in the finalmap. Feature extraction from digital imageryis, in general, a complicated problem, so thatthe computer is often unable to accomplishthe task alone. Therefore, it appears reason­able to combine the superior pattern recog­nition abilities of the human being with thecomputational power of the digital computerto form an interactive system. In this paperwe are concerned with the extraction offeatures by the interactive digital imagesystem in order to generate photo coordi­nates automatically. If we have the photocoordinates of features from stereo photogra­phy, we can use analytical photogrammetricmethods to generate ground coordinates andto compile line maps.

INTERACTIVE DIGITAL IMAGE

PROCESSING SYSTEM

An interactive digital system offers flexi­bility. Once the basic hardware and softwaresystems are developed, only minimal soft­ware effort is required to utilize the systemfor new applications. The interactive digitalimage processing system which was utilizedto conduct this research was developed atthe U.S. Army Engineer Topographic Labo­ratories, Fort Belvoir, Virginia. The hard­ware consists of a CDC-6400 sequentialcomputer, a Goodyear STARAN associativearray processor, a DEC PDP 1l/50 mini­computer, standard peripherals, and asso­ciated image display, digitizing, and record­ing devices. The CDC 6400, STARAN, andPDPll computers have their own systemsoftware package to handle all operations fortheir system. There are interface softwarepackages to allow all computers in the sys­tem to communicate with each other.

Images to be processed on the interactivedigital image processing system are input tothe system from magnetic tape or digitizedtransparencies made on the DICOMED056. Once the images are in the system,they are stored on CDC's 844 disks. Theinteractive capability is achieved by usersworking through the Tektronix 4014-1 alpha-

numeric console and the COMTAL imagedisplay unit. The computations are done inthe CDC-6400 computer and STARAN par­allel processor that allows the programmerto take advantage of parallelism in the arith­metic structure of a problem. The finalresults can be put on hard copies, printed ontransparencies on-line by the DICOMEDD47, or off-line by copying to magnetic tapefor recording on other hard copy devices.

BASIC ALGORITHMS

A typical picture usually consists primar­ily of features having homogeneous densi­ties and certain properties, such as size,shape, direction, orientation, and adjacencywhich are most impOitant for map makers. Ifwe can accurately determine the boundaries(edges) of the objects, we may be able tomeasure these essential properties. We as­sume that the boundary around an object is aclosed loop except as it reaches the limits ofthe data. The following algorithms to extractthe features by using the interactive digitalimage systems have been developed.

EDGE DETECTION

Initially, the user has to provide a startingpoint and a threshold value for detectingedges after selecting a feature for manipula­tion. The basic approach that we have de­signed consists of two steps. The first isfinding an initial point. The computer com­pares the gray level of one pixel at a time ofa small array, whose upper-left corner is thegiven starting point, against the input thresh­old value. If the threshold gray level lies be­tween the gray levels of two adjacent pixels,an edge exists between the two pixels. If thetwo adjacent points are on the same line, theorientation of the edge is vertical across thelength of the pixel. If the two adjacent pointsare on the different lines, the orientation ofthe edge is horizontal.

The second step is detecting a new edgepoint by comparing the gray levels of threenearest neighbors of the initial point againstthe threshold value. It is done according tothe orientation (horizontal and vertical) ofthe edge. Suppose (see Figure 1) a horizon­tal edge point is at location @ betweenelements (i, j) and (i +1, j) and assume thegray level of element (i, j) ~ T (the thresholdvalue) and the gray level of element (i +1, j)~ T. If the edge has an extension, there arethree cases; the new edge point may be(a) between elements (i, j) and (i, j +1),(b) between elements (i, j +1) and (i +1,j +1), or (c) between elements (i+ 1, j +1)and (i + 1, j). Only two neighbor elements

FEATURE EXTRACTION BY INTERACTIVE IMAGE PROCESSING 1497

op j+1

~I• p

+ + + + +

+ a .1 +--~ II b

;+1 c .

R.

FIG.!. Detecting a new edge point from a knownhorizontal edge.

need be checked to determine the locationof the edge. If the gray level of element (i,j +1) ",:; T, it is case a. If it is not case a, andthe gray level element (i +1, j +1) ",:; T, it iscase b; if the gray level of element (i +1,j +1) > T, it must be case c.

If the previous edge is vertical and the lastedge point is at location ® (see Figure 2)and assuming the gray level of element (i, j);;" T and the gray level of element (i, j +1)",:; T. There are three cases. The edge may be(a) between the elements (i, j) and (i +1, j),if the gray level of element (i +1, j) ",:; T; (b)between the element (i + 1, j) and (i + 1,j +1), if it is not case a and the gray level ofelement (i +l,j +1) ",:; T; and (c) between theelement (i, j+l) and (HI, j+l), if it isneither case a nor case b.

LINE DETECTION

A line is a long and narrow region whichhas a certain width, homogeneous gray lev­els in the region, and a gray level distinctly

op

;+1

R.FIG. 2. Detecting a new edge point from a knownvertical edge.

different from its background. It is difficultto detect a line by the edge detecting meth­od· but if we can determine the width andth~ central points of the line, we may beable to measure the essential properties.Initially the user must provide a startingpoint, the critical value (Ie) of the gray leveldifference between adjacent elements in theline region, after he has selected a linefeature for manipulation. There are twosteps for detecting. The first step is search­ing all parameters which are required in thesecond steps, such as the width, direction,orientation, and the gray level of the line.The second step is locating the centralpoints of the line by comparing the differ­ence between the gray level of elements andthose of the background against the criticalvalue (IG) and also comparing the differenceof gray level between adjacent elementsalong the line against the critical value (Ie)according to the orientation of parameter ofthe line. If the difference of the average graylevel of the cross section of the line and theaverage gray level of background is greaterthan the critical value (IG) and the differ­ence of the average gray level of two adja­cent cross sections of the line is smaller thanthe critical value Ie, then the new points arethe line points and the central point of thecross section is recorded.

PROCEDURES

A stereo pair of aerial photographs #98and #99 over the Arizona Test Area wereused. The scale of photography is about1:47,100. The Model1050A Microdensitom­eter system was used to measure and scanthe images for this research. A point in theupper-left corner of the photo was chosen asthe origin of the photo coordinates system.For both photos, all fiducial marks, thereseau intersections of interest, and theupper-left corner of the mapping area whichwould be digitized were measured with thescanner in the manual mode, and the photocoordinates were recorded. An aperaturesize of 34.5 micrometers and the 5x magni­fication for the eyepiece and the objectivelens were chosen. Then scan line and pixelspacings were selected to be 30 micrometerseach. A square array (2048 x 2048 points) oneach photo was digitized, and the data wererecorded on magnetic tapes. After inputtinga portion of the digital data from the photog­raphy into the system and displaying it onthe screen, the user can examine the sceneand decide which feature will be extractedfirst. Then the subset of data which con-

1498 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING, 1977

(b) If it is case b, the location of the newedge point is between element (i + 1,j) and (i+ 1, j + 1); then the localcoordinates of the point are

(c) If it is case c, the location of theedge point is between the elements(i+ 1, j + 1) and (i, j + 1); then thelocal coordinates of the point are

(c) If it is case c, the location of the newpoint is between element (H 1, j)and (H 1, j + 1); then the local coor­dinates of the point are

p = op + j + 0.5 * IFG

op and ol are the coordinates of theupper-left corner point of the datamatrix. IFG is the indicator of edgedirection. If IFG=I, the edge isfrom left to right. If IFG= -1, theedge is from right to left. The unit ofthe p and l is the pixel spacing of thedigitized data.

(5)

(4)

(3)

l = ol + i + 1.

l=op+i+1.

p = op + j + 0.5

l = ol + i + 0.5 * IFG.

and

and

and

For detecting an edge point from a knownvertical edge and a threshold value, the prin­Cipal detecting method and procedure arethe same as the known horizontal edge. Thedifference is the orientation of the knownedge; therefore, the comparison is done~ounterclockwiseand the ways for generat­mg the local coordinates are different.

The local coordinates of edge points arecalculated as follows (see Figure 2):

(a) If it is case a, the location of the newedge point is between element (i, j)and (i + 1, j); then the local coordi­nates of the point are

p=op+j

tained the feature will be moved into thecomputer and displayed on the screen at alarger scale.

EDGE DETECTION

If an edge is to be detected, the usershould input a starting point and a thresholdvalue. The starting point can be entered intwo ways: one is the coordinates of thepoint, the other is the position of the cursoron the screen. There are three ways toobtain the threshold value: (1) Users willprovide the threshold value which must beinput to the system; (2) the program willprovide the threshold value by using themean gray level of the data; or (3) by plottingthe histogram of the gray levels of the dataand using the bottom value of the valley ofthe histogram.

The first step is to search for an initialedge point in a small array in which theupperleft corner is the given starting point.The computer compares the gray level ofone pixel at a time against the thresholdvalue. If the threshold gray level lies be­tween the gray levels of two adjacent pixels,an edge exists between the two pixels. Afterfinding an edge point, the program will stopthe search and record the local coordinatesof the initial edge point for the edge and theorientation of the edge. The local coordi­nates of the initial edge point (refer again toFigures 1 and 2) are p = op + j +0.5 andl = ol + i for the vertical edge and p = op + jand l = ol + i +0.5 for the horizontal edge,respectively.

The second step is to locate the edgepoints according to the orientation of theedge. If there is a horizontal edge, the localphoto coordinates of the edge points arecalculated as follows (see Figure 1):

(a) If it is case a, the location of the newedge point is between elements (i,j) and (i, j + 1); then the local coordi­nates of the point are

p = op + j + 0.5 * IFG

and (1)

l = ol + i.

(b) If it is case b, the location of the newedge point is between element (i,j+I) and (HI, j+I); then the localcoordinates of the point are

p=op+j+I p=op+j+I

and (2) and (6)

l = ol + i + 0.5. l = ol + i + 0.5 * IFG.

FEATURE EXTRACTION BY INTERACTIVE IMAGE PROCESSING 1499

IW is the width of the line and g(i, k) is theleft nearest background neighbor of the linesection.

Equations 8 and 9 must be satisfied simul­taneously. The cross section at i is a partof the line and the central point is at g(i, k+ (IW/2)). If some cross section fails theEquation 8 test and passes the Equation 9test, the program will not register the crosssection as a part of the line and will processthe next cross section until some sectionfails both tests, then the program terminatesthe line.

The program will generate the local coor­dinates for the central points of a line byusing Equation 10.

where ap and 01 are the local coordinates ofthe upper-left corner of data matrix, and IWis the width of line.

The principles of and procedures for de­tecting other orienting lines are the same asthose for the vertical line. The only differ­ence is in the equations for testing andcalculating the local coordinates; therefore,that the details are omitted.

All edge or line points are displayed onthe screen by an overlay. If no additionalnew points for the edge or line can be lo­cated, or the edge or line has reached thelimit of the data, the process is paused.Operator assistance is needed to give a newstarting point or perform other operations. Ifa boundary of an edge or line is terminated,the local coordinates can be output. How­ever, because the edges or lines are usuallysmooth curves, the two-dimensional raw

(10)

IW

I = 01 + i

p = ap + k + IW/2

op;'-_-..£::.::::::::::::~---+p

01 0 x x:x x x 0

0 x x x x x 0

g(; ,k(~x x@x x 0

x x x x x 0

0 x x x x x 0

0 x x x x x 0

0 x x x X x 0

0 x x x x x 0

0 x x x x x 0

0 x X x x x 0

0 x x x x x 0

0 x x x x x 0

0 x x x x x 0

0 x x x x x 0

1

FIG. 3. The vertical line.

and

ap and 01 are the coordinates of theupper-left corner point of the datamatrix. IFG is the indicator of edgedirection; if IFG = 1, the edge isfrom top to bottom; if IFG = -1, theedge is from bottom to top. The unitof the p and I is the pixel spacing ofthe digitized data.

IW (9)-I g(i+l,j+k)/ ,;;;IC

j ~ 1

LI E DETECTION

If a line is to be detected, the user shouldprovide a starting point and two criticalvalues (IG and Ie). The program will firstsearch the area around the starting point togenerate the property parameters such aswidth, direction, orientation, and the initialcentral point of the line by counting theelements whose difference of gray levels issmaller than the critical value IC. If thehorizontal length of the region is longer thanthe vertical length, the line is orientatedhorizontally and the vertical length is thewidth of the line; the middle point of thecross section is the central point of the line.If the horizontal length is equal to thevertical length, it means the line is a 45­degree tilted line. The parameters for thetilted line are obtained in the same manneras for the horizontal or vertical line. Afterthe program has all parameters, it detectsthe line according to its orientation. If it is avertical line, the average gray level of thehorizontal elements of the line (the crosssection of the line) is compared to the aver­age of two background neighbors. If thedifference of the average gray level of theline and average gray level of the back­ground is greater than the critical value IGand the difference of the average gray levelof the line and average gray level of the back­ground is greater than the critical value IGand the difference of the average gray level oftwo adjacent cross section of line is smallerthan the critical value IC, then the newpoint is the line point (see Figure 3). Themathematical expressions are as follows:

1 IW

1- I g(i,j+k)IW j = 1

and

- ~ [g(i, k) + g(i, k+IW)J I ~ IG (8)

1 IWIrw I g(i,j+k)

j = 1

1500 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING, 1977

data (local coordinates) might be smoothedas a straight line or a second-order curveaccording to their shape by a least squareadjustment program. After adjustment, theresults are displayed on the screen; then theuser may decide how many points will beused and/or stored as the local coordinatesfor representing the feature of the photo­graph.

The local coordinates of correspondingfeature points can be obtained by using adigital image match technique. An indepen­dent array (11 x 11) whose central point isthe known feature point will move aroundon the corresponding image to find a match.The correlation values are computed accord­ing to the following equation for matching:

opPo£

6P~

6£

1- + + +£0

+ 0-, +0d

+ + +

£

FIG. 4. Two dimension correlation function andthe match point.

PHOTO COORDINATES

The photo coordinates of features aregenerated:

GGAT (m, n) (11)

=II(AD *TD )/(VAR[A] *VAR[T]lij

where A is the dependent array, T is theindependent array,

and

x = ULX + p * s - ox

(13)

ij

and

AD = [A(i, j) - AM],TD = [T(i - m,j - n) - TM],

Var [A] II (AD)2,

y = -(VLY + 1* s - oy).

Where ULX and VLY are the plate coordi­nates of the upper-left corner ofthe digitizedarea; p and 1 are the local coordinates offeature; s is the scale factor (the pixel spac­ing); and ox and oy are the plate coordinatesof the principal point. The photo coordinatesof the features accompanying descriptivetitle or heading on each file and a series ofcodes for each line segment are a part of theoutput. The codes will indicate the type andkind of features. One code will indicatewhether the feature is an edge or a line. If itis a line, the width of the line should berecorded, since the line detecting routineonly gives the coordinates of the centralline. If it is a closed boundary, a codeindicating the texture within the region isneeded. Another code will indicate whetherthe data represent a straight line or a curve.If it is a curve, the mathematical model isused for the least square adjustment routine.From the codes, the map compiler can deter­mine precisely the characteristics of thefeature without referring to the photograph.

(12)

I' = 1 + !i.l + ff.1.

p' = p + !i.p + ff.p

and

Var [T]

Var [A] and Var [T] are variances of A and T,and AM and TM are the mean of A and T,respectively.

The routine then finds the maximum valueamong the correlation values and its eightnearest neighbors, and it fits a second-ordersurface through the nine points by a leastsquare adjustment routine. The adjustedpeak of the surface is the matched point (seeFigure 4).

The coordinates of the matching point are:

where p and 1 are estimated coordinates, !i.pand !i.1 are shifts between the estimated pand 1 coordinates of the largest computedcorrelation location, and ff.P and ff.l are thecorrections of coordinates from the leastsquares adjustment. p' and I' are the finallocal coordinates for the corresponding fea­ture points.

CONCLUSION

The tracing of features manually in greatdetail is time-consuming and requires skilledoperators. Since these tasks are routine,repetitive operations that require great pre­cision, it would be desirable to automate theentire process. However, many of these

FEATURE EXTRACTION BY I TERACTIVE IMAGE PROCESSING 1501

tasks require perceptual abilities and areexpected to continue to be beyond the ca­pabilities of a fully automatic system for thenear future.

An alternative approach is to combineabilities of man and machine-an interactivesystem. This approach has several advan­tages.

• Decisions can be made and adapted to theproblem as frequently as the operator likes.When errors are committed, he may inter­rupt the process to make corrections and/orto improve the results until he is satisfied.

• Techniques can be selected depending onthe context. If a routine procedure does notwork in a particular situation, the operatormay try various techniques to process thedata. This flexibility gives researchers avery useful tool for experimenting withvarious techniques and procedures for eval­uating data for exploratory development.

• Only the data of immediate interest to theuser will be handled. As a result, less spacewill be required and computer time willbe reduced. The computations which areinherently parallel in nature were per­formed on the STARAN parallel processorto speed up the processing.

This paper demonstrates the capability forextraction of cartographic detail from digitalaerial photographic data. It appears to offera substantial improvement in speed over themanual method. Also, its interactive capa­bility between operator and computer offersa great potential for applications in otherfields.

REFERENCES

1. Crombie, Michael A., Semiautomatic PassPoint Determination Using Digital Tech­niques, ETL-005I, U.S. Army Engineer Top­ographic Lab., Ft. Belvoir, VA 22060, De­cember 1975.

2. Gambino, L. A. and M. A. Crombie, "DigitalMapping and Digital Image Processing,"Photogrammetric Engineering, November1974.

3. Ketting, R. L. and Lundgrebe, AutomaticBoundary and Sample Classification of Re­motely Sensed Multispectra Data, LARSInformation Note 041773, LARS PurdueUniversity, April 1973.

4. VandeBrug, G. J., "Line Detection in SatelliteImage," Machine Processing of RemotelySensed Data Symposium, Proceedings, LARSPurdue University, June 1975.

Call for papers

Seventh AnnualRemote Sensing of Earth Resources Conference

Tullahoma, TennesseeMarch 27-29, 1978

Sponsored by the University of Tennessee Space Institute, the technical program willfeature papers reflecting recent developments in

• Remote Sensing Techniques• Land Use Planning• Exploration and Management of Natural Resources• Agricultural Surveys and Planning• Forest Inventory• Air and Water Pollution Measurement and Analysis• Wetlands and Soil Moisture Mapping• Instrumentation and Sensor Equipment• Photo Interpretation and Photogrammetric Applications

Those wishing to present papers please submit the title of the proposed paper; the author'sname, address, and position; and a 140-word abstract by January 10, 1978 to

Dr. F. ShahrokhiConference DirectorThe University of Tennessee Space InstituteTullahoma, Tennessee 37388