IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-5, NO. 2, MARCH 1983

at points u = ±1/N, we must have

R*(u)--sup Rff(u)) = Ico[- +2u

(A1 8)

The points 1/N are anomalous for the reason that our cover-ing of the interval [- 2, -] did not include the end point x =Now in ordinary integration (Reimann-Lebesque) a single pointcannot affect the values of an integral and so, for example, wemust have R(l) = 0. But if "mass points" are allowed (e.g.,Stieljes integration), then a single point can influence an in-tegral. In such case, (Al 8) is correct for all u and, for example,

R*(1) = cos--3 2

is correct. But for ordinary integration

R * (-) = cosN N + I

Inequality 3: Let f, denote the restriction of f to x < -A, f2the restriction to -A S x 6 A, and f3 the restriction to x > A.Let r1 denote the energy in fl, r2 in f2 and r3 in f3. Then byassumption we must have

r, + r2 + r3 = 1

f = fl +f2 +f3* (A19)Using the notation

fu fu(x) f(x - u)we have

R(u) = R(u) = (f fuf)ff

= (fl +f2 +f3,fl;u +f2;u +f3;u)

subject to

r, +r3= r.

(The value of the maximum is easily shown to be (2 - r)r).The case u > 2A is obtained in a similar mannerInequality 4: The basic method is similar to that for inequal-

ity (3). Decompose g as follows

g=gl +g2 (A23)where

g, = R*f(x + u*)

g2 =g-g1-

Apply inequality (I) to the term (f, gl;u) and use the fact that

J+g2 dx= 1 - R *.

_0

Inequality 5: By inequality (4)

R(u)<R*c+ l R*2

but also

R(u) S R *

Hence

R(u) S min {R*, R*a + 1F }R*}.Maximize the RHS of the above to get inequality (5).

REFERENCES[11

[2]

(A20) [3]If we expand (A20) there formally appear nine terms. How-ever, by symmetry we may assume u > 0, and this causes threeof these terms to vanish on account of their having disjointsupport. (For example (fA, f2;u) = 0.) The surviving termsmay be written

[4]

[51

(A24)

(A25)

(A26)

R. 0. Duda and P. E. Hart, Pattern Classification and Scene Anal-ysis. New York: Wiley, 1973, pp. 305, 319.A. Rosenfeld and A. C. Kak, Digital Picture Processing. NewYork: Academic, 1976, pp. 18, 414-416.A. Gelb, Ed., Applied Optimal Estimation. Cambridge, MA: MITPress, 1974, pp. 36-38.B. P. Lathi, Signals, Systems and Communication. New York:Wiley, 1965, pp. 515-522.A. K. Jain, "Image coding via a nearest neighbor image model,"IEEE Trans. Commun., vol. COM-23, pp. 329-331, Mar. 1975.

R(u) = (f2,f2;u) + (f3,f2;u +f3;u)+ (fl +f2, fl;U') + (f3,fl;u)-

(A2 1)

Assuming that u < 2A the last term in the above also vanishes.Apply inequality (1) to the first term and apply the Cauchy-Schwarz inequality to the remaining two terms to get

Solving Problems in Robotics with Semantic Networks

KRISHNA K. AGARWAL

iftr[2A

+ v,/r NV 1+ V, fl

-1+2

IuI

According to the statement of inequality (3) we have

r, +r3 =r

r2 = 1 - r.

The desired conclusion now follows by maximizing

-V/ 3 + I V 7r

Abstract-Robot problems are examined in the context of semanticnetworks which are used to represent the state of a problem and the

r3. operators useful for solving it. Graph transformation algorithms arediscussed as an aid to problem solving. Although these form only asmall subset of the first-order predicate calculus based systems, con-

(A22) siderations such as subgoal circularity, partially specified states andmultiple manipulators sharing the same environment may warrant thissimplification.

Index Terms-Backwards analysis, graph transforms, planning, problemsolving, robotics, semantic networks.

Manuscript received October 20, 1980; revised September 17, 1982.The author is with the Department of Computing and Information

Science, Trinity University, San Antonio, TX 78284.

0162-8828/83/0300-0213$01.00 1983 IEEE

IR(u)I6 r2 COS

213

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-5, NO. 2, MARCH 1983

I. INTRODUCTIONIn recent years it has become clearer that robot construc-

tion involves two independent components-a logical solutionformulator and an electromechanical task executor. It has alsobeen observed that problems in these two areas are much harderthan previously believed. Not only is it difficult to write pro-grams which can analyze a problem, it is also an extremelycomplex task to discover its solution. Once a plausible solu-tion is obtained, its execution by means of electromechanicaldevices is certainly nontrivial and may or may not be success-ful. If unsuccessful, the solution formulator must be reinvokedfor an auxiliary plan. If properly designed, the solution formu-lator has the advantage of knowing why the original plan didnot succeed.Stanford University has done pioneering work in robotics

with their problem-solver STRIPS [6]. STRIPS can solve avariety of problems dealing with an environment of rooms,switches, doors and boxes for the robot. First-order predicatecalculus is used to represent the state of the robot's world, theoperators which change the state, checks for inconsistent states,and rules for logical implicants. As in GPS [5], "means-endsanalysis" is used to conduct a heuristic search for a solution,i.e., a sequence of operators which may produce a transforma-tion of the initial state to the final state.

It is clear that predicate logic is not the only mechanism forsolving such problems. It does provide "completeness" [ 11 ],i.e., theoretically it is possible to represent and solve any prob-lem within its framework. On the other hand, one of its short-comings involves the difficulty in determining whether or nottwo formulas represent the same state. This determination isessential to delete circular subgoals during the problem solvingphase. Further problems arise if states are only partially speci-fied. The theorem prover may be unable to make the decisionsnecessary to proceed with the problem solving process. Whentwo or more robots work simultaneously in the same environ-ment, it becomes difficult to generate a plan where they allcollaborate to solve the problem.The framework for the problem affects the search for a solu-

tion [4], [10], [11], [14]. Several problems can be repre-sented by semantic networks or labeled graphs [ 14]. We feelthat these networks can be used not only to represent certainproblems, but also to solve them. Graph embedding and trans-formation algorithms [1], [2] permit us to create a dynamicenvironment for problem solving.By keeping the representation straightforward, it is hoped

that the problems of subgoal circularity, incompletely speci-fied states and multiple manipulators will become more

manageable.

1I. A ROBOT PROBLEM

Fig. 1 illustrates a problemn which is difficult for a robotto solve, although it does not appear to be so because of the"natural way" a person can solve it.The problem may be simply stated as: "turn on the light in

room R 1." Several assumptions have been made in the prob-lem. It is assumed that the robot can move between rooms, itcan turn the light switch ON or OFF, the switch activates thebulb, the robot can detect whether or not the bulb is lit, it canremove the bulb from its socket if necessary, it can pick up abulb from the shelf in room R 2, etc. Some of these assump-tions have been made more explicit in the semantic network ofFig. 2. The problem is simply restated as: "the final state mustcontain the network of Fig. 3 as a subgraph."

Fig. 4. illustrates the operators required to solve this prob-lem. To decrease the number of invalid subgoals generated,

Socket

Switch (off)

Bi

B2,B3,B4

I 1000 1Rl R2

R1,R2 ---- RoomsBl,B2,B3,B4 ---- Bulbs

Fig. 1. Can the robot turn on the light in room RI?

c --- "connected to by a door"i --- "is inside of"t --- "turns on"h --- "holds"o --- "placed on"

Fig. 2. Representation of the initial state using semantic networks.

Fig. 3. Subgraph desired in the fmal state.

simple rules can be applied before and after an operator is ap-plied. For example, before the robot moves to an adjacentroom by using the operator 01, it can be assumed to be no

longer adjacent to any other object. Similarly, after the robotturns on the switch by using operator 03, the state should bemodified to indicate whether the bulb lights up. Hence, a

postoperation transform may have to be applied.Fig. 5 illustrates a plausible solution which could be obtained

214

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-5, NO. 2, MARCH 1983

app-yi 01

Before apyn or 01 repeatedly apply:

Before applying 04, test for appearance of:

al--- adjacent to01, 01 : Robot can go to an adjacent room.

Rx 02 ( )i i 1

02-

Before applying 02, repeatedly apply:

02:1 Go to Switch, Socket or Shelf in the room.02 : Move away from Switch, Socket or Shelf.

After applying 04, apply (to dispose bad bulb):

Q0 ~~~~~0Before applying 04 , ascertain that the following is absent:

haa

After applying 04 if bulb lights up, apply:

After applying 03, if bulb lights up, apply:

h

is ies

04:1 Remove bulb from socket.04 : Put bulb in socket.

POST-03Bx

isPOST-03 1

LIT

03:1 Turn ON switch.03 : Turn OFF switch.

05 1 Pick up bulb from shelf.05 : Place bulb on shelf.

Fig. 4. Operators available to the robot.

by conducting a backwards analysis [7], [81, [10], [131. Ateach stage we check the subgoals for validity. The analysis iscontinued until we reach a state (termed a "plausible" initialstate) which is such that it violates no conditions of the knownportion of the initial state. After a level of subgoal generation,we must select one of the "open" subgoals for further analysis.Techniques such as the "uniform-cost" search method [10]can be utilized for this purpose. However, a heuristically com-puted cost must be associated with each subgoal before any ofthese methods can be successfully used.Note that there is no guarantee that this solution will cause

the bulb to light because it may be burnt out. If so, the stateSO' of Fig. 6 will be obtained after the sequence 01, 02, 03has been attempted. At this stage, forward analysis may beconducted from SO' to SO (or another backwards analysis fromSO to SO'). In either case, Fig. 7 illustrates another sequencewhich could be used if SO' is reached. Finally, Fig. 8 illustrates

the entire procedure developed by piecing together all thestraightline sequences developed by the problem solver.The introduction of multiple robots has no effect on the

representations of the states and the operators. Parallel opera-tion of the robots can be simulated during the problem solvingphase by permitting the simultaneous application of one ormore operators as long as there is no "interference" betweenthe multiple applications of the operators. Similar techniqueshave been used in the simultaneous multiple application of achemical transform during the synthetic planning for organicmolecules [ 1 ], [2] . The basic features of these techniques arethe detection of "active" edges in each transform and the re-jection of a multiple application if two or more active edgesembed on to the same edge of the state undergoing the trans-formation. States which may not be obtainable by successiveapplications of the operators may be produced during simul-taneous applications.

215

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-5, NO. 2, MARCH 1983

Rl c R2

iAi

t

Socket witch

h is

Bx ON

is

LIT

S0 (part of the final state)

Q31

1. Go to room Rl (Apply 01).

2. Go to switch (Apply 02).

3. Turn switch ON (Apply 03).4. If there is light, stop.

5. Go to socket (Apply 02).6. Remove and dispose bulb (Apply 04).7. Go to room R2 (Apply 01).8. Go to shelf (Apply 02).9. If no bulb on shelf, stop with failure.

10.

11.

12.Socket

Rl c R2

i

i

t

Socket 4 SwitchRh is

Bx OFF

is

NOT-LIT

NOT-LIT

011

"A 1 E c-1R

i

Socket t-witch i

R

h is

Bx OFF

is

NOT-LIT

Fig. 5. A plausible solution.

Fig. 6. The plausible solution may result in this state.

1. Go to socket (Apply 02).

2. Remove and dispose bulb (Apply 04).

3. Go to room R2 (Apply 01).

4. Go to shelf (Apply 02).

5. Pick bulb from shelf (Apply 05).

6. Go to room Rl (Apply 01).

7. Go to socket (Apply 02).

8. Put bulb in socket (Apply 04 ).

Fig. 7. Another plausible solution.

Pick bulb from shelf. (Apply 05).Go to room Rl (Apply 01).Go to socket (Apply 02). -

13. Put bulb in socket (Apply 04 ).

14. If bulb is lit, stop.

15. Continue at step 6.

Fig. 8. The entire solution.

III. CONCLUSIONSReal life problems must be examined in the context of se-

mantic networks and related problem solving techniques.Problems such as assembly line operations, fault testing andautomatic programming may become more manageable viathese methods.Many problems must be solved before a plan can be success-

fully executed. For instance, the robot must be able torecognize boxes and other objects it must deal with. Here toowe feel that semantic networks can be used in conjunctionwith graph embedding techniques [31, [12] since objects canbe represented as graphs whose nodes represent lines and sur-faces and whose edges represent their relationships. As theplan is executed by the robot, it must be able to recognize thestates on the solution path so that it may apply the next oper-ator in the plan. The problem solving phase requires the de-velopment of graph transformation algorithms such as thosediscussed in [1], [21. Libraries of operators have to be con-structed. The backwards and forwards analysis problem solvingprocesses have to be formulated. Heuristics for subgoal selec-tion must be discovered before a wide variety of problems canbe handled.To enhance the power of the problem solving system, we

must be able to "learn" by saving solutions as more powerfuloperators, a technique that is expected to be similar to theconstruction of "macrops" in STRIPS (6], [9].Simultaneous operation of several cooperating robots work-

ing on a partially specified environment is too difficult a taskfor present day problem solvers, most of which are basedon the first-order predicate calculus. Perhaps the simplifica-tion of the representation will make this realistic task moremanageable.

Finally, it is our opinion that more unconventional techniquesmust be applied if progress is to be more rapid. Is it necessaryto build robots which, by simulating humans, perform taskssimilarly, or is it worthwhile providing the robot with better re-mote control and remote sensing capabilities? We feel thatgraph embedding and transformation algorithms are very power-ful tools and their problem solving abilities are yet to be realized.

REFERENCES

[1] K. K. Agarwal, "Transformation and canonization algorithms forgraph representable structures with applications to a heuristicprogram for the synthesis of organic molecules," State Univ.New York, Stony Brook, Tech. Rep. 63, 1976.

[2] K. K. Agarwal, D. L. Larsen, and H. L. Gelernter, "Application ofchemical transforms in SYNCHEM2; A computer program fororganic synthesis route discovery," Computers and Chemistry,vol. 2, pp. 75-84, 1978.

216

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-5, NO. 2, MARCH 1983

[31 K. K. Agarwal, D. P. Agrawal, and M. L. Lassner, "An algorithmfor subgraph identification," in Proc. Princeton Univ. Conf. In-form. Sci. and Syst., Mar. 1980.

[4] A. Bundy, R. M. Burstall, S. Weir, and R. M. Young, ArtificialIntelligence: An Introductory Course. North-Holland, 1978.

[51 G. W. Ernst and A. Newell, GPS: A Case Study in Generality andProblem Solving (ACM Monograph). New York: Academic, 1969.

[6] R. E. Fikes and N. J. Nilsson, "STRIPS: A new approach to theapplication of theorem proving in problem solving," ArtificialIntell., vol. 2, pp. 189-208, 1971.

[71 H. L. Gelernter, "Realization of a geometry theorem provingmachine," in Computers and Thought. New York: McGraw-Hill, 1963.

[8] H. L. Gelernter, A. F. Sanders, D. L. Larsen, K. K. Agarwal, R. H.Boivie, G. A. Spritzer, and J. E. Searleman, "Empirical explora-tions of SYNCHEM," Science, Sept. 9, 1977.

[9] E. B. Hunt,ArtificialIntelligence. New York: Academic, 1975.[101 N. J. Nilsson, Problem Solving Methods in Artificial Intelligence.

New York: McGraw-Hill, 1971.[11] B. Raphael, The Thinking Computer-Mind Inside Matter. Free-

man, 1976.[12] A. F. Sanders, "Some applications of graph theory to the design

of a heuristic program for the discovery of organic synthesis,"Ph.D. dissertation, Dep. Comput. Sci., State Univ. New York,Stony Brook, 1976.

[13] J. R. Slagle, Artificial Intelligence: The Heuristic ProgrammingApproach. New York: McGraw-Hill, 1971.

[14] P. H. Winston, Artificial Intelligence. Reading, MA: Addison-Wesley, 1977.

An Intelligent Tactile Sensor-An On-Line HierarchicalObject and Seam Analyzer

SASA PRESERN AND LUDVIK GYERGYEK

Abstract-This paper describes an on-line computer supported tactilesensor that can recognize complex industrial objects, identify a seamlocation, and track a three-dimensional seam trajectory for an arc weld-ing robot. A one finger tactile sensor with two degrees of freedom isused. The supporting microcomputer system organizes and reduces theuseful sensor data from digitized tactile information to a compact sym-bolic representation which is matched to a knowledge network. Abeamsearch algorithm is used. The hierarchical organization provides thatsimple features are first detected on the object, and more complexfeatures are examined later. Pruning of the search tree is based on alikelihood function.

Index Terms-Artificial intelligence, object recognition, on-line seamtracking, tactile sensor.

INTRODUCTIONRecently many approaches for seam tracking [1], [21 and

tactile sensing [3], [4] have been proposed. The system de-scribed in this paper is a model-based tactile system for objectidentification and seam tracking in industrial robotics. Anintelligent tactile sensor is used for recognition of simple andcomplex curved three-dimensional objects and tracking of athree-dimensional seam trajectory. It is designed for applica-tion in industrial robotics for arc welding where informationabout the object is needed, and the use of a computer visionsystem is not suitable for various reasons (e.g., the object is

Manuscript received August 25, 1982; revised November 16, 1982.The authors are with the Jozef Stefan Institute, Jamova 39, Ljubljana,

Yugoslavia.

not in view of a camera, or very dusty, polluted, or hostileenvironments).An intelligent tactile sensor, which was developed in our

laboratories (dimensions 15 X 4 X 4 cm), is a one finger sensorwith two degrees of freedom (Fig. 1) [5], [6]. The maximaldisplacement of a measuring needle at the bottom of the sensoris - 10 mm to +10 mm in each direction. Two masks with graycode are fixed at the top of the measuring needle, one for eachdegree of freedom. The position of each mask is optically re-corded by a set of six phototransistors, and therefore the tac-tile information is a 2 X 6 array of binary data. Information isprocessed from sensors to a Z80 microcomputer. The sensingsystem, with feedback control, has a resolution of 0.05 cm.The distance between the welding torch and the mounted tac-tile sensor can be adjusted over a range of 3 to 10 cm.A new technique for tactile sensing requires a new approach:

formulation of a knowledge base, definition of primitive objectelements, and effective matching strategies.

II. SYMBOLIC DESCRIPTION AND DATABASEREPRESENTATION

In order to recognize an object or the location of a seam, itis desirable to have symbolic descriptions of the spatial proper-ties of the object. Such a description, if properly associatedwith the corresponding part of the tactile strategy, greatlysimplifies the representation of the object, and leads to fastermatching and search analysis.The spatial data of an object are organized and reduced in

stages. The database system for object recognition and seamidentification by an intelligent tactile sensor is organized as acollection of hierarchical descriptions where each level in thehierarchy represents a different conceptual abstraction of theobject geometry.

First the tactile signal is used to detect the most character-istic locations on the surface of the object. Later detailed"touching" of specific locations on the surface is performed.This step of organizing and reducing the object data is knownas signal-to-symbol transformation. All these symbolic descrip-tions represent each object by a list of properties and form adatabase of a tactile sensing system.The following list of properties is used to describe objects:

Xi(X)MXi(-x)Yi(y)MYi(-y)Zi(z)Hi(x, y, z; r)

is the +x extreme of an object i at (y = O,z = 0)is the -x extreme of an object i at (y = 0, z = 0)is the +x extreme of an object i at (x = 0, z = 0)is the -y extreme of an object i at (x = 0, z = 0)is the +z extreme of an object i at (x = 0, y = 0)is the hole at location x, y, z with diameter r.

Symbolic descriptions of objects are stored in a long term data-base and have the form of simple expressions, such as

H4(x,y, z;r)

01 62-8828/83/0300-0217$01.00 © 1983 IEEE

217