Robust estimation of surface curvature from deformation

of apparent contours

Andrew Blake and Roberto Cipolla

apparent contour to the intrinsic curvature of the

surface (Gaussian curvature); the sign of Gaussian

curvature is equal to the sign of the curvature of the

contour. Convexities, concavities and inflections of an

apparent contour indicate. respectively. convex. hypcr-

bolic and parabolic surface points. Giblin and Weiss’

have extended this by adding viewer motions to obtain

quantitative estimates of Gaussian and mean curvature.

A surface can be rcconstructcd from the envelope of all

its tangent planes, which in turn are computed directly

from the family of silhouettes of the surface. obtained

under planar motion of the viewer. By assuming that

the viewer follows ;I grtwt circle of \kwcr directions

around the object they restricted the problem ot

analysing the envelope of tangent planes (a

Z-parameter family) to the less general one: of comput-

ing the envelope of a family of lines in ;I plane. Their

algorithm was tested on noise-free. synthetic data (on

the assumption that extrcmal boundarics had been

distinguished from other image contours) dcmonstrat-

ing the reconstruction of ;I planar curve under ortho-

graphic projection.

I‘his paper cxtencis these theories further. to the

general USC of arbitrarv non-planar camera motion

under perspective projection. The Gaussian curvature

of ;I surface. at a point on its silhouette. can be

computed given some known local motion of the

vicwcr. C’urv:lture is computed from \p~ltio-tempor~ll

derivatives (up to $;ccond order) 01 image-rneasur~ll~~~

quantities. The theory can. ol COIII-se. be applied to

detect extremal boundaries and distinguish them from

sur-i’acc markings 01. discontinuitie~. Experiments show

that. with adcquatc vicwcr-m<,tion calibration, itsell

computed from L isual d;rt;i I. ir is possible lo obtain

cur\ ature nieasuremcnts of u\efuI ;Icc’ur;ic\.

A co~~scqucnce of the theory. representing an important step towards qudlitativC vision. concerns the

robustness of rtdlrti\v or t/(/7i~rcvr/irrl mt’asurcments ot

curvature at two ncarbl points. Intuitively it is rcla-

tivcly difficult to judge moving around ;I smooth,

featureless obJect, whether its silhoutttc is extremal OI-

not -~ whether the Gaussian cur\,aturc along the

contour is bounded or not. This judgement is much easier to make for objects with feature-rich surfaces. Under small viewer-motions, features are ‘sucked’ over the extremal boundary, at a rate which depends on surface curvature. Our theory reflects this intuition exactly. It is shown that relative measurements of curvature across two adjacent points are entirely immune to uncertainties in the viewer’s rotational velocity. This is somewhat related to earlier results showing that relative measurements of this kind are important for depth measurement from optic flow”, and for curvature measurements from stereoscopically viewed highlights’. Furthermore, they are relatively immune to uncertainties in translational motion in that, unlike single-point measurements, they are indepen- dent of the viewer’s acceleration. Only dependence on velocity remains. Experiments show that this theo- retical prediction is borne out in practice. Differential or relative curvature measurements prove to be more than an order of magnitude less sensitive than single- point measurements to errors in viewer-motion calibra- tion. There is some theoretical evidence that ratios of differential curvature measurements are less sensitive. In our experiments absolute measurements of curva- ture were so sensitive that they became unreliable for viewer motion errors of 0.5mm in position and lmrad in orientation. For ratios of differential measurements of curvature the corresponding figures were about 50 mm and 70 mrad respectively.

THEORETICAL FRAMEWORK

Surface geometry

Consider a point P on the extremal boundary of a smooth, curved surface in R” and parameterized locally by a vector valued function r(s, t). The parametric representation can be considered as covering the surface with two families of curves: r(s, to), and r(sO, t) where sa, to are fixed for a given curve in the family. A one parameter family of views is indexed by the time parameter t and s, tare defined so that the s-parameter curve r(s, to) is an extremal boundary for a particular view to. A t-parameter curve r(so, t) can be thought of as the 3D locus of points grazed by a light-ray from the viewer, under viewer-motion. Such a locus is not uniquely defined.

Local surface geometry can be specified in terms of the basis {rS, r,} for the tangent plane (r, and r, denote drlas and &Vat, respectively) and the surface normal - a unit vector n.

Imaging model

The imaging model is a spherical pin-hole camera of unit radius. The image of the world point P with position vector r(s, t) is a unit vector T(s, t) defined by:

r(s, t) = v(t) + hT(s, t) (1)

where A is the distance along the ray to the point P (see Figure 1).

For a given vantage position to the apparent contour is a continuous family of rays T(s, to) emanating from the camera’s optical centre which touch the surface so that T. n = 0. The moving observer at position v(t) sees a two parameter family of apparent contours T(s, t).

108

Figure I. Surface and viewing geometry. P lies on a smooth surface which is parameterized by r(s, t). For a given vantage point v(to) the family of rays emanating from the viewer’s optical centre (C) that touch the surface defines an s-parameter curve r(s, to) - the extremal boundary from vantage point to. The spherical perspective projection of this extremal boundary - the apparent contour T(s, to) - determines the direction of rays which grazes the surface. The distance along each ray CP is A. A moving observer at position v(t) sees a two parameter family of extremal boundaries r(s, t) whose spherical perspective projections are represented by a two parameter family of apparent contours T(s, t). t-parameter curves @(so, t) and T(s,, t)) are not uniquely defined

Properties of the extremal boundary and its projection

For perspective projection we derives the following well-known properties of the extremal boundary and its projection’~3~9~10:

The orientation of the surface normal n can be recovered by measuring the direction of the ray T of a point on an extremal boundary and the tangent to the apparent (image) contour T,:

Tr\T,

“=m (2)

The ray direction T and the tangent to the extremal boundary r, are in conjugate directions with respect to the second fundamental form:

T.n,=O (3)

The ray direction and the extremal boundary will only be perpendicular if the ray T is along a principal direction. The curvature of the apparent contour K” (more precisely the goedesic curvature of the curve T(s, to) which can be computed from spatial derivatives in image measurements), can be written in terms of the normal curvature of the extremal boundary K~:

image and vision computing

Equations (4) and (5) show that an apparent contour is smooth except for a special viewing geometry when the ray direction runs along the extremal boundary. At such points the apparent contour may have a cusp. For opaque surfaces only one branch of the cusp is visible, however, corresponding to a contour- ending’,“.

Choice of parameterization

There is no unique spatio-temporal parameterization of the surface. The mapping between extremal boundaries at successive instants is undetermined. The problem of choosing a parameterization is an ‘aperture problem’ for contours on the spherical perspective image (T(s, t)) or on the Gauss sphere (n(.s. t)), or between space curves on the surface r(s, t). A natural parameterization is the ‘epipolar parameterization’ defined by:

r, T=O (6)

For this parameterization the tangent-plane basis vectors r, and r, are in conjugate directions (from equation (3)).

Differentiating equation (1) with respect to time, and enforcing equation (6) leads to the ‘matching’ condi- tion*:

T, = (v, T) T

A (7)

Points on different contours are ‘matched’ by moving along great-circles on the image sphere with poles defined by the direction of the viewer’s instantaneous translational velocity v,. This induces a natural corres- pondence on the surface between extremal boundaries from different viewpoints. If the motion is linear corresponding points on the image sphere will lie on an cpipolar great-circle. This is equivalent to epipolar plane matching in stereo. For a general motion, however. the epipolar structure rotates continuously as the direction of v, changes and the space curve r(s,,, t) generated by the movement of a contact point will be non-planar.

The parameterization will be degenerate when {r,,, r,} fails to form a basis for the tangent plane. This occurs if the contour is not an extremal boundary but a 3D rigid space curve (when r,=(l), or at a cusp/ contour-ending in the projection (when r,h,r, = 0, see ~thove”. The parameterization degrades gracefully and hence these conditions pose no special problems.

Information available from the deformation of the apparent contour

WC have shown’ that local surface geometry can be recovered from spatio-temporal derivatives (up to 2nd order) of image measurable quantities and known

If WC choow the rclcrcncc frame to hc the instantaneous camera co-trrdlnatc sydcm WC can cxprc~s T and T, in terms (II a spherical Image position vector Q. and image vclocitics (optic flow) Q,. Namely. T = Q and T, = Q, + co Q. Equation (7) rcducc\ to the equation of motton and structure from motion from optic Ilou I’,

viewer motion. We summarize the most important results below.

1. Recovery of depth Depth (distance along the ray A) can be computed from the rate of deformation (T,) of the apparent contour under known viewer motion (translational velocity) v,)“. The depth is given by:

v, n A=-- -

T, n (8)

This formula is an infinitesimal analogue of triangula- tion with stereo cameras. The numerator is analogous to baseline and the denominator to disparity. The result also holds for a rigid space curve or an occluding boundary. It is independent of choice oi parameterizationx.

2. 1,ocal surface curvuturr The normal curvature at Pin the direction of the ray T. K’ (which is the same as the normal curvature of the space curve r(q), t)) can be computed from the rate of deformation (T,) of the apparent contour under viewer motion, and its temporal derivative. This requires knowledge of viewer motion (translational and rota- tional velocity and acceleration):

(T,. n)-’

K’= (T,,.n)(v,.n)+Z(T.v,)(T,.n)‘--(v,,.n)(T,.n)

The sign and magnitude of Gaussian curvature can then be computed from the product of the normal curvature

I K , and the curvature of the apparent contour K”

(measured by equation (4)) scaled by inverse-depth’:

(10)

EXPERIMENTAL RESULTS: DETERMINING CURVATURES FROM ABSOLUTE MEASUREMENTS

Figure 2 shows three views from a sequence of a scene taken from a camera mounted on a moving robot-arm whose motion has been accurately calibrated from visual data’. Using a numerical method for estimating surface curvatures from three discrete views (set Reference 8) we can estimate the radius of curvature of the norma/ secfion R (where K’ = l/R) for a point on an extremal boundary of a cup B. The method is repeated for a point which is not on an cxtremal boundary but is a surface marking A. This is a degenerate cast‘ of the parameterizntion. A surface marking can be considered as the limiting case of a

point with infinite curvature and hence ideally will have zero ‘radius of curvature’. If the measurements are error-ridden and the motion is not known accurately, however. surface markings will appear as extrcmal boundaries on surfaces with high curvature.

The radius of the cup was measured using calipers as 43.4 312 mm. The estimated curvatures agree with the actual curvatures. However. the results are very

IOY

a b

Figure 2. Estimating surface curvatures from three discrete views. Points are selected on image contours in view I (a), indicated by crosses A and B for points on a surface marking and extremal boundary respectively. For epipolar parameterization of the surface corresponding features lie on epipolar lines in views 2 and 3 (2b and 2~). Measurement of three view vectors lying in an epipolar plane can be used to estimate surface curvatures

Table 1. Radius of curvature (of normal section defined by the ray direction) estimated from three distinct views of a point on a surface marking and a point on an extremal boundary

Surface marking A Extremal boundary B

Mcasurcd Actual

(mm) (mm)

I.05 0.0 45.7 44.4

Error

(mm)

I .YS 1.3

sensitive to perturbations in the motion parameters (see Figures 3a and 3b).

DIFFERENTIAL MEASUREMENT OF CURVATURE

We have seen that although it is perfectly feasible to compute curvature from the observed deformation of an apparent contour, the result is highly sensitive to motion calibration errors. This may be acceptable for a moving camera mounted on a precision robot-arm or when a grid is in view so that accurate visual calibration can be performed. In such cases it is feasible to determine motion to the accuracy of around 1 part in 1000 that is required. However, when only crude estimates of motion are available another strategy is called for. It is sometimes possible in such a case to use the crude estimate to bootstrap a more precise visual egomotion computation”. However, this requires an adequate number of identifiable corner features, which may not be available in an unstructured environment. Moreover, if the estimate is too crude the egomotion computation conditioned14.

may fail; it is notoriously ill-

The alternative approach is to seek qualitative measurements of geometry that are much less sensitive to error in the motion estimate. In this section we show that relative or differential measurements of curvature have just this property. Differences of measurements at two points are insensitive to errors in rotation and in translational acceleration. Typically, the two features might be one point on an extremal boundary and one

110

fixed surface point. The surface point has infinite curvature and therefore acts simply as a stable refer- ence point for the measurement of curvature at the extremal boundary. Intuitively, the reason for the

a

/ 1

B

A

position error/nlm

b B

Figure 3. Sensitivity of curvature estimated from abso- lute measurements to errors in motion. The radius of curvature (mm) for both a point on a surface marking (A) and a point on an extremal boundary (B) is plotted against error in the estimate of position (a) and orientation (b) of the camera for view 2. The estimation is very sensitive and a perturbation of I mm in position produces an error of 190% in the estimated radius of curvature for the point on the extremal boundary. A perturbation of I mrad in rotation about an axis defined by the epipolar plane produces an error of 70%

image and vision computing

insensitivity of differential curvature is that global additive errors in motion ineasurement are cancelled out I

Consider two visual features whose projections on the image sphere are T(s,, r), i= 1, 2 which we abbreviate to T’, i = I . 2. Think of them as two points on extrcmal boundaries. which trace out curves with (normal) curvatures K” and K” as the viewer moves. The first temporal derivatives of T’ are dependent only on image position, viewer velocity and rotation and depth:

T; = (v, ‘I”) T’

/\ (11)

Second order temporal derivatives are:

+ 3(Tr. v,)(T:. n) -v,, . n 1 i=1,2 (12)

Let us define two relative quantities. The differrnriul curvature AK’ of the feature pair is defined by:

I I I ---__=__-_ AK’ K/l K/2 (13)

Note that it is not an infinitesimal quantity but a difference of inverse curvature. The relative view vectcw is defined to be:

6(t) = T(s?. 1) - T(.s,, t) (14)

Consider the two features to be instantane~3usly spatially coincident, i.e. initially T(s,, t) = T(s,,t). Moreover. assume they lie at a common depth A, and hence, instantaneously, T?; = T:. In practice, of course. the feature pair will only coincide exactly if one of the points is a surface marktng which is instantaneously on the cxtremal boundary. Now. taking the difference of equation (12) for i = 1, 2 leads to a relation between these two tiiJftwnfia1 quantities:

(13

From this equation, we can obtain dijferentiul curvnturl~ AK’ as :I ~uI~cti(3n of depth A, viewer velocity v,. and the second ttmporal derivative of 6. Absolute mea- suremcnt of curvature (12) depended also on the viewer’s translational acceleration v,,. Uncertainty from practical measurements (based on finite differences. for example) of the lower derivative will. of course. be much reduced. Hence, the relative measurement should be much more robust. Moreover, because 6 is a relative mcasuremcnt on the projection sphere. unlike the image vectors T’ which occur in the absolute nle~sLlr~rnerlt of curvature, it is un~lffected by errors in vicwcr rotation.

In the case that T’ is known to be a fixed surface reference point, with I/K” = 0, then AK’ = K” so

that the differential curvature AK’ constitutes an

a

estimate, now much more robust, of the normal curvature K” at the extrcmal boundary point ‘I“. <It course. this can now be used in cyuation (10) to obtain :I robust estimate of Gaussian curvatlirc.

Our experiments confirm this. Figures 41 and Ib show that the sensitivity of the differential curvature to error in position and rotation computed between points A and E3 (two nearby. points at similar depths) is reduced by an order ot magnitude. ‘l‘his is iI striking decrease in sensitivity even though the features do not coincide exactly as the theory rcquircd.

Further robustness can be obtained by considering the ratio of ~~ifferenti~l curvatures. Ratios of two-point differential curvaturt‘ mcasurcmcnts are, in tlieorv. complctelv insensitive to bicwcr motions. ‘l‘his -is because in a rfltio ot’ ki measurements for two rf{fffcwnt point-pairs, terms dcpcnding on ahsolutc depth h and velocity v, are cancelled out in equation (15). This result corresponds to the following intuitive idea. The rate at which surface fcaturcs rush towards or away from an extremul boundary is proportional to the (normal) curvature there. The constant of propor- tionality is some fuI~cti(~ll of vic~ver-Iil(~ti(~li and depth; it can be eliminated bv considering only ratios of curvatures. Results (SW Figure\; 5;1 and 3) show another striking decrcasc in aensiti\itv - of another order of magnitude.

7

Figure 5. Sensitivity of ratio of differential curvatures. The ratio of differential curvatures measurements made between two points on an extremal boundary and the same nearby surface marking is plotted against error in the position (a) and orientation (6) of the camera for view 2. The sensitivity is further reduced by an order of magnitude to 1.5% error for a I mm error in position and I. 1% error for I mrad error in rotation. The vertical axes are scaled by the actual curvature for comparison with Figures 3 and 4

CONCLUSION

We conclude that in theory and practice given just one surface reference point, highly robust relative curva- ture measurements can be made at points on apparent contours. Moreover, the technique can achieve by motion analysis something which has so far eluded photometric analysis: namely discrimination between fixed surface features and points on extremal bound- aries.

We are currently working on the realtime imple- mentation of the theories presented in this paper. Preliminary results involving the tracking of image contours using deformable contours (snakes); detection of extremal boundaries and the fast, robust recovery of strips of surface in the vicinity of an extremal boundary are described in Reference 15.

ACKNOWLEDGEMENTS

The authors acknowledge discusions with Professor Mike Brady, Dr Andrew Zisserman, Dr Peter Giblin, Dr David Murray and Dr David Forsyth. We acknow- ledge the support of the IBM UK Scientific Centre. the SERC and EEC.

REFERENCES

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Barrow, H G and Tenenbaum, J M Intrinsic Scene Characteristics from

Recovering Images AI

. . Center Technical Report 157, SRI International, USA (1978) Koenderink, J J ‘What does the occluding contour tell us about solid shape?’ Perception Vol 13 (1984) pp 321-330 Giblin, P and Weiss, R ‘Reconstruction of surfaces from profiles’ Proc. 1st Int. Conf. Comput. Vision London, UK (1987) pp 136-144 Tsai, R Y ‘A versatile camera calibration techni- que for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses’ IEEE J Robot. & Automat. Vol 3 No 4 (1987) pp 323-344 Longuet-Higgins, H C and Pradzny, K ‘The interpretation of a moving retinal image’ Proc. R. Sot. Lond. B Vol 208 (1980) pp 385-397 Weinsball, D Direct computation of 30 shape and motion invariants AI Memo 1131, MIT, USA (1989) Blake, A and Brelstaff, G ‘Geometry from specu- larities’ Proc. 2nd Znt. Conf. Comput. Vision (1988) pp 394-403 Blake, A and Cipolla, R Robust Estimation of Surface Curvature from Deformation of Apparent Contours Technical Report OUEL 1787/89, University of Oxford, UK (1989) Koenderink, J J and Van Doorn, A J ‘The shape of smooth objects and the way contours end’ Percep- tion Vol 11 (1982) pp 129-137 Brady, M, Ponce, J, Yuille, A and Asada, H ‘Describing surfaces’ Comput. Vision, Graph. & Image Process. Vol 32 (1985) pp 1-28 Belles, R C, Baker, H H and Marimont, D H ‘Epipolar-plane image analysis: an approach to determining structure’ Int. .I. Comput. Vision Vol 1 (1987) pp 7-55 Maybank, J J ‘The angular velocity associated, with the optical flow field arising from motion through a rigid environment’ Proc. R. Sot. Lond. A Vol 401 (1985) pp 317-326 Harris, C G ‘Determination of ego - motion from matched points’ 3rd Alvey Vision Conf. Cambridge, UK (1987) pp 189-192 Tsai, R Y and Huang, T S ‘Uniqueness and estimation of three-dimensional motion para- meters of rigid objects with curved surfaces’ IEEE Trans. PAM1 Vol 6 No 1 (1984) pp 13-26 Cipolla, R and Blake, A The dynamic analysis of apparent contours Technical Report OUEL 18281 90, University of Oxford, UK (1990)

112 image and vision computing