Gait Recognition using Dynamic Affine Invariants

Alessandro Bissacco† Payam Saisan‡ Stefano Soatto†

† Computer Science Department ‡ Electrical Engineering DepartmentUniversity of California, Los Angeles University of California, Los Angeles

Los Angeles, CA 90095 Los Angeles, CA 90095

Abstract

We present a method for recognizing classes of human gaitsfrom video sequences. We propose a novel image basedrepresentation for human gaits. At any instance of time agait is represented by a vector of affine invariant moments.The invariants are computed on the binary silhouettes cor-responding to the moving body. We represent the time tra-jectories of the affine moment invariant vector as the out-put of a linear dynamical system driven by white noise. Theproblem of gait classification is then reduced to formulatingdistances and performing recognition in the space of lineardynamical systems. Results demonstrating the discriminatepower of the proposed methods are discussed at the end.

1 Introduction

We live in a dynamic world, constantly analyzing and pars-ing time varying streams of sensory information. Almostall biological creatures equipped with the sense of visionuse dynamic cues to analyze their surrounding for criticalsurvival decisions. Clearly there is an abundance of infor-mation embedded in the dynamics of visual signals1. In thiswork we focus on extracting and exploiting the temporalstructure in video sequences for the purpose of recognizinghuman gaits.

Observing a person walking from a distance, we can of-ten tell whether the subject is a human, identify their gen-der, or make predictions about individual traits like age orphysical health. We postulate that such information is en-coded not necessarily in the static appearance, but mostlyin the dynamicsof the moving body. In Johansson’s ex-periments [34] one cannot tell much from a single frame,however when the sequence is animated suddenly the sceneis easily parsed. Johansson’s experiments show that even inthe lack of all comprehensible static content, the dynamicsof a few moving dots can contain sufficient information to

1Naturally there is also a great deal of information in the photometryand geometry of the scene that can be conveyed in a single static frame.However, in this study we concentrate on the scene dynamics.

correctly decipher the underlying physical phenomenon. Inthis paper we address the problem of recognizing a personwalking from one jumping, runnning, hopping or dancing,independent on the person and his pose.

1.1. Previous Work

The problem of image-based human motion analysis andrecognition has been receiving considerable attention in theliterature. Most of the proposed approaches involve track-ing the pose of the human body, represented either as kine-matic chain of body parts [11, 16, 21], or as spatial arrange-ment of blobs [9] or point features [1]. Statistical mod-els, such as standard [5, 1] and parametric [6, 17] HiddenMarkov Models are then fitted to the tracking data and like-lihood tests are used for recognition. In [10, 12, 2] mixed-state statistical models for the representation of motion havebeen proposed, and in [18, 19] particle filters have been ap-plied in this framework for estimation and recognition. In[4] linear gaussian models have been used, and recognitionis performed by defining a metric on the space of models.

Other techniques do not require an explicit model of thehuman body. Zelnik-Manor et al. [15] propose a statisticsof the spatio-temporal gradient at multiple temporal scalesand use it to define a distance between video sequences.

Some approaches [14, 20, 8] are specific to recognitionof periodic motion, such as the human gaits we consider inthis paper. In [14] classification is based on periodicitiesof a similarity measure computed on tracked moving parts.Little and Boyd [8] use Fourier analysis to compute the rel-ative phase of a set of features derived from moments ofoptical flow, and employ the resulting phase vector for clas-sification. Bobick and Davis [7] propose a description basedon the spatial distribution of motion, the Motion Energy andMotion Histogram Images. Recognition is done by compar-ing Hu moments [22] of of those images with a set of storedmodels. In [13], the problem is recognizing actions fromvideo taken from a distance, where the person appears onlyas a small patch. They compute a set of spatio-temporal mo-tion descriptors on a stabilized figure-centric sequence, andmatch the descriptors to a database of preclassified actions

1

using nearest neighbohr classification.

2 Extracting an Affine InvariantRepresentation

An instance of a gait here is an image sequence of abouttwo to three seconds (50-70 frames) long. It contains a hu-man subject performing an action like walking or running.In their raw pixel form image sequences are far too clut-tered with irrelevant information. We are only interestedthe part related to the person in the image and more specifi-cally his/her motion. A successful isolation of the dynamicsinformation is highly dependent on the extraction of a repre-sentation that is insensitive to such nuisance factors like thebackground, clothing, lighting and viewing angle. Whilewell established techniques with arbitrary degrees of so-phistication can be deployed for extracting appearance freerepresentations we note that simple silhouette’s are conve-niently insensitive to appearance factors1. They can be eas-ily extracted from motion sequences using background sub-traction techniques. However, we need to be able to rec-ognize a walk not just invariant to appearance, but also toviewing geometry. Much work has been done with fea-tures like textures, edges, transform coefficients (Fourier,wavelets) and matrix factorizations. To account for per-spective distortions and variations of the viewing geome-try we must go beyond these and consider more generalstatistical features. While an appearance free feature wasstraightforward to attain, extracting a geometric invariantfeature requires some attention. For this we follow well es-tablished results from theory of geometric invariance andlook at affine invariance. Utility of affine invariance is re-alized by the fact that general affine deformations can helpaccount for a range of perspective distortions. Specificallywe are looking for scalar featuresFi’s (working on silhou-ettes) that are invariant to general affine transformations, i.e.F {I(u, v)} = F {I(x, y)}, where[

uv

]=

[a1 a2

a3 a4

] [xy

]+

[b1

b2

]A concise and elegant development of affine invariants

based on higher order central moments is discussed in [23].Flusser et al, begin with the assumption that the affine in-variant can be expressed in terms of the central momentsof the binary image. Drawing from the theory of algebraicinvariants, they use two-dimensional moments of the imageto derive explicit expressions for independent affine invari-ants. Here the general two dimensional (p+q)’th order cen-tral moments of an ImageI(x, y) are defined as :

1Other solutions such as threshholding the optical flow, or working di-rectly with optical flow magnitude produced almost identical results.

µp,q =∫∫

(x− x)p(y − y)qI(x, y)dxdy

The x andy are the coordinates of the center of gravityof the image. An invariant F is assumed to have the form ofa polynomial of the central moments :

F =∑

i

ki

∏j

µpj(i),qj(i)

/µz(i)00

We include here the final form of the expressions for thefirst four invariants, and refer the reader for derivations to[23]:

I1 =(µ2,0µ0,2 − µ2

1,1

)/µ4

0,0

I2 =(µ2

3,0µ20,3 − 6µ3,0µ2,1µ1,2µ0,3 + 4µ3,0µ

21,2

+ 4µ2,1µ20,3 − 3µ2

1,2µ22,1

)/µ10

0,0

I3 =(µ2,0

(µ2,1µ0,3 − µ2

1,2

)µ2

0,3 − µ1,1 (µ3,0µ0,3 − µ2,1µ1,2)

+ 3µ0,2

(µ3,0µ1,2 − µ2

2,1

))/µ7

0,0

I4 =(µ3

20µ203 − 6µ2

20µ11µ12µ03 − 6µ220µ02µ21µ03

+9µ220µ02µ

212 + 12µ20µ

211µ21µ03

+6µ20µ11µ02µ30µ03 − 18µ20µ11µ02µ21µ12

−8µ311µ30µ03 − 6µ20µ

202µ30µ12 + 9µ20µ

202µ

221

+12µ211µ02µ30µ12− 6µ11µ

202µ30µ21

+ µ302µ

230

)/µ11

00

The idea of using moments on motion region is not new.In [7] Hu moments are used on a description of the spatialdistribution of motion for recognition of activities. How-ever, Hu moments are invariant only under translation, ro-tation and scaling of the object. By using the momentsproposed in [23], we obtain a representation of the movingshape invariant to general affine transformations.

It should be noted that moment invariants are particu-larly natural for binary silhouettes. Furthermore the invari-ance to translation eliminates the need for tracking people,body parts or blobs; a common preprocessing scheme ingait modelling.

In this section we outlined a representation that is simplebut powerful. We will use this descriptor in the next sectionsisolate the dynamics of a gait from its image sequences. Wewill then discuss how to go from time trajectories of fea-tures to dynamical models and cast the gait classification asrecognition in the space of linear dynamical systems.

2

3 Dynamic modelling with Invariants

We make the assumption that temporal behavior of theinvariant as the gait evolves in time, can be sufficientlyrepresented as a realization from a second-order station-ary stochastic process. This means that the joint statisticsbetween two instants is shift-invariant. This is a restric-tive assumption that will allow for modelling of station-ary gaits and not for “transient” actions. It is well knownthat a positive definite covariance sequence with rationalspectrum corresponds to an equivalence class of second-order stationary processes. It is then possible to chooseas a representative of each class a Gauss-Markov model -that is the output of a linear dynamical system driven bywhite, zero-mean Gaussian noise - with the given covari-ance. In other words, we can assume that there exists apositive integer , a process (the “state”) with initial condi-tion x0 ∈ Rn ∼ N (0, P ) and a symmetric positive semi-

definite matrix

[Q SST R

]≥ 0 such that{y(t)} is the

output of the following Gauss-Markov “ARMA” model2:{x(t + 1) = Ax(t) + v(t) v(t) ∼ N (0, Q); x(0) = x0

y(t) = Cx(t) + w(t); w(t) ∼ N (0, R)(1)

for some matricesA ∈ Rn×n andC ∈ Rm×n.The first observation concerning the model (1) is that

the choice of matricesA,C,Q,R, S is not unique, in thesense that there are infinitely many models that give riseto exactly the same measured covariance sequence startingfrom suitable initial conditions. The first source of non-uniqueness has to do with the choice of basis for the statespace: one can substituteA with TAT−1, C with CT−1,Q with TQTT , S with TS, and choose the initial condi-tion Tx0, whereT ∈ GL(n) is any stablen × n matrixand obtain the same output covariance sequence; indeed,one also obtains the same output realization. The secondsource of non-uniqueness has to do with issues in spectralfactorization that are beyond the scope of this paper [28].Suffices for our purpose to say that one can transform themodel (1) into a particular form – the so-called “innova-tion representation” – that is unique. In order to be able toidentify a unique model of the type (1) from a sample pathy(t), it is therefore necessary to choose a representative ofeach equivalence class (i.e. a basis of the state-space): sucha representative is called acanonical model realization(orsimply canonical realization). It is canonical in the sensethat it does not depend on the choice of the state space (be-cause it has been fixed).

While there are many possible choices of canonical real-izations (see for instance [29]), we are interested in one thatis “tailored” to the data, in the sense of having a diagonal

2ARMA stands for autoregressive moving average.

state covariance. Such a model realization is calledbal-anced[30]. The problem of going from data to models thenbe formulated as follows:given measurements of a sam-ple path of the process:y(1), . . . , y(τ); τ >> n, estimateA, C, R, Q, a canonical realization of the process{y(t)}.Ideally, we would want the maximum likelihood solutionfrom the finite sample, that is the argument of

maxA,C,Q,R

p(y(1), . . . , y(τ)|A,C, Q,R). (2)

The closed-form asymptotically optimal solution to thisproblem has been derived in [31]. From this point on, there-fore, we will assume that we have available – for each sam-ple sequence – a model in the form{A,C,Q,R}. While thestate transitionA and the output transitionC are an intrinsiccharacteristic of the model, the input and output noise co-variancesQ and R are not significant for the purpose ofrecognition (we want to be able to recognize trajectoriesmeasured up to different levels of noise as the same). There-fore, from this point on we will concentrate our attention onthe matricesA andC that describe a gait.

4 Recognizing gaits

Models, learned from data as described in the previous sec-tion, do not live on a linear space. While the matrixA isonly constrained to be stable (eigenvalues within the unitcircle), the matrixC has non-trivial geometric structure forits columns form an orthogonal set. The set ofn orthogo-nal vectors inRm is a differentiable manifold called “Stiefelmanifold”.

Because of the highly curved structure of this space,state-of-the-art classification algorithms applied on themodel parameters fail to produce satisfactory results. Inparticular, we tested an efficient implementation [25] of theSupport Vector Machines classifier [24] on the vectors ob-tained by stacking the elements of the matricesA andC.With this approach discrimination was not possible even inthe simple case of only two classes of gaits.

4.1 Distance between models

As proposed in[4], a natural solution for the recognitionproblem in this case is provided by endowing the space ofmodels with a metric structure. In the literature of systemidentification and signal processing, the problem of defin-ing a metric in the space of linear dynamical systems is anactive area of research [26, 27]. A common distance thatis widely accepted in system identification for comparingARMA models is based on the so-called subspace angles[31].

3

Given a modelM specified by the matrices(A,C), theinfinite observability matrixO(M) is defined as:

O(M) =[

CT AT CT A2T CT · · ·]∈ R∞×n

The matrixO(M) spans an n-dimensional subspace ofR∞. To compare two modelsM1 andM2, the basic ideais to compare “angles” between the two observability sub-spaces ofM1 and M2. There are many equivalent waysto define subspace angles. Given a matrixH with itscolumns spanning an n-dimensional subspace, letQH de-note the orthonormal matrix which spans the same subspaceas H. Given two matricesH1,H2, we denote the n or-dered singular values of the matrixQT

H1QH2 ∈ Rn×n to be

cos2(θ1), ..., cos2(θn). Then the principle angles betweensubspaces spanned byH1 and H2 are denoted by the n-tuple:

H1 ∧H2 = (θ1, θ2, ..., θn) , θi ≥ θi+1 ≥ 0.

Based on these angles, two distances can be defined:

d2M = − ln

∏i

cos2(θi), dF = θ1. (3)

The first distance is an extension of the Maritin distancedefined for SISO systems [27], the second is the Finsler dis-tance according to Weinstein [32]. Roughly speaking, thedifference between these two distances is thatd2

M is anL2-norm butdF is anL∞-norm between linear systems.

Once a metric in the space of models is available, stan-dard grouping techniques such as k-means clustering can besuccessfully employed for recognition.

5 Experiments and Results

Our gait dataset consists of short clips of walking (with andwithout a backpack), limping, running and jumping per-formed by two subjects, for a total of 81 sequences. InTable 4.1 we show a more detailed description of the exeri-mental data, and in Figure 1 sample frames from the videosequences.

Given a gait sequence, for each frame we used back-round subtraction to extract a sihouette of the moving body,and computed the affine invariant moments on this binaryimage. Figure 2 shows sample output of the backgroundsubtraction, and in Figure 4.1 the trajectories of the mo-ments for some sequences in the dataset are plotted. Fromthe experiments, we noticed that moments of order higherthan4 are too sensible to noise and negatively affect the re-sults. Also, the four momentsI1, I2, I3, I4 have differentscales and need be normalized to form the feature vectory(t). The values of the scale factors were found empirically.

Figure 1:Sample frames from the dataset of the gaits: walk-ing, running, jumping and limping.

Figure 2:Sample silhouettes extracted by background sub-traction.

For each sequence of moment trajectoriesy(t) we haveidentified a dynamical model of ordersn = 1 to 4. Foridentifying the model we used the implementation of theN4SID algorithm [33] in the Matlab System IdentificationToolbox. Since our models are zero-mean, we subtract themean from the data before the learning step.

We have then computed the mutual distance betweeneach model by calculating the distances between observ-ability subspaces: the Finsler distancedF and our general-ization of the Martin distancedM , as defined in(3). Thesetwo distances gave similar results, with an advantage forthe latter one. In figure 5 we show the pairwise distance be-

4

AW

alk

1AWalk1

AW

alk

2AWalk2

AW

alk

3AWalk3

AW

alk

4AWalk4

AW

alk

5AWalk5

AW

alk

6AWalk6

AW

alk

7AWalk7

AW

alk

8AWalk8

AW

alk

9AWalk9

AW

alk

10

AWalk10A

Walk

11

AWalk11A

Walk

12

AWalk12A

Walk

13

AWalk13A

Walk

14

AWalk14A

Walk

15

AWalk15A

Walk

16

AWalk16A

Walk

17

AWalk17A

Walk

18

AWalk18A

Walk

19

AWalk19A

Walk

20

AWalk20B

Walk

1BWalk1

BW

alk

2BWalk2

BW

alk

3BWalk3

BW

alk

4BWalk4

BW

alk

5BWalk5

BW

alk

6BWalk6

BW

alk

7BWalk7

BW

alk

8BWalk8

BW

alk

9BWalk9

BW

alk

10

BWalk10B

Walk

11

BWalk11B

Walk

12

BWalk12B

Walk

13

BWalk13A

Walk

Bkpk1

AWalkBkpk1A

Walk

Bkpk2

AWalkBkpk2A

Walk

Bkpk3

AWalkBkpk3A

Walk

Bkpk4

AWalkBkpk4A

Walk

Bkpk5

AWalkBkpk5A

Walk

Bkpk6

AWalkBkpk6A

Walk

Bkpk7

AWalkBkpk7A

Walk

Bkpk8

AWalkBkpk8A

Run1

ARun1A

Run2

ARun2A

Run3

ARun3A

Run4

ARun4A

Run5

ARun5A

Run6

ARun6A

Run7

ARun7A

Run8

ARun8A

Run9

ARun9B

Run1

BRun1B

Run2

BRun2B

Run3

BRun3B

Run4

BRun4B

Run5

BRun5B

Run6

BRun6B

Run7

BRun7B

Run8

BRun8B

Run9

BRun9B

Run10

BRun10B

Run11

BRun11A

Jum

p1

AJump1A

Jum

p2

AJump2A

Jum

p3

AJump3A

Jum

p4

AJump4A

Jum

p5

AJump5A

Jum

p6

AJump6A

Jum

p7

AJump7A

Jum

p8

AJump8A

Jum

p9

AJump9B

Jum

p1

BJump1B

Jum

p2

BJump2B

Jum

p3

BJump3B

Jum

p4

BJump4A

Lim

p1

ALimp1A

Lim

p2

ALimp2A

Lim

p3

ALimp3A

Lim

p4

ALimp4A

Lim

p5

ALimp5A

Lim

p6

ALimp6A

Lim

p7

ALimp7

Figure 5:This is the distance grid based on subspace distances between observability matrices.

tween models of sequences in the dataset, with highlightedthe two nearest neighbohrs.

As the result show, the dynamics of the invariant is ableto distinguish between different sytles of gaits while pre-serving the invariance with respect to apperance and geo-metric factors. Discrimination fails only when comparingsequences of walking and limping, due to the high similar-ity of these two classes of motion.

6 Conclusions

We introduced a method to incorporate both appearance andgeometric affine invariances to represent gait sequences.We discussed how to extract such invariant representationsfrom images and model their temporal behavior. We thendeveloped a frame work where invariant representationscould be modelled and compared in the space of linear dy-namical systems. We presented results using over 100 se-quences of gait samples corresponding to various different

5

0 10 20 30 40 50 60 70−0.02

−0.01

0

0.01

0.02

0.03

0.04I1I2I3I4

0 5 10 15 20 25 30 35 40−0.02

−0.015

−0.01

−0.005

0

0.005

0.01

0.015

0.02

0.025I1I2I3I4

Figure 3: Plots of affine invariant moments computed onthe binary silhouettes: on the left moments from a walkingsequence of subject A, on the right moments from a runningsequence of subject B.

Gait Number of SequencesSubject A Subject B

Walking 20 13Walking with backpack 8 0

Running 9 11Jumping 9 4Limping 7 0

Figure 4:Description of the gait dataset: 4 gait classes per-formed by 2 persons for a total of 81 sequences, details asabove.

viewing angles and gait types. We were able to cluster andisolate sequences corresponding to same or similar gaits re-gardless of the subject or geometric viewing factors. Thepotential and power of affine invariant representation anddynamical systems as a descriptor for the dynamic structureof gaits is clear and present from the results. This appearsto be promising front that will be be explored further.

References

[1] Y. Song, X. Feng, and P. Perona, Towards detection of humanmotion. InProc. of CVPR, pages 810-817, 2000.

[2] C. Sminchisescu and B. Triggs. Kinematic Jump Processesfor Monocular Human Tracking. InProc. of CVPR, 2003

[3] D. M. Gavrila. The visual analysis of human movement: Asurvey. InComputer Vision and Image Understanding, vol-ume 73, pages 82–98, 1999.

[4] A. Bissacco, A. Chiuso, Y. Ma and S. Soatto. Recognitionof Human Gaits. In Proc. of the IEEE Intl. Conf. on Comp.Vision and Patt. Recog., pages 401-417, December 2001.

[5] T.Starner and A. Pentland. Real-time american sign languagerecognition from video using hmm. InProc. of ISCV 95, vol-ume 29, pages 213–244, 1997.

[6] A. D. Wilson and A. F. Bobick. Parametric hidden markovmodels for gesture recognition. InIEEE Trans. on Pattern

Analysis and Machine Intelligence, volume 21(9), pages 884–900, Sept. 1999.

[7] A. F. Bobick. and J. W. Davis The recognition of humanmovement using temporal templates. InIEEE Trans. PAMI,23(3):257-267, 2001.

[8] J. J. Little and J. E. Boyd. Recognizing people by their gait:the shape of motion. 1996.

[9] C. R. Wren and A. P. Pentland. Dynamic models of humanmotion. InProceedings of FG’98, Nara, Japan, April 1998.

[10] C. Bregler. Learning and Recognizing Human Dynamics inVideo Sequences. InProc of CVPR, pp. 568-574, 1997

[11] C. Bregler and J. Malik Tracking People with Twists andExponential Maps InProc. of CVPR, 1998

[12] V. Pavlovic and J. Rehg and J. MacCormick. Impact of Dy-namic Model Learning on Classification of Human MotionIn Proc. of International Conference on Computer Vision andPattern Recognition, 2000.

[13] A. A. Efros, A. C. Berg, G. Mori and J. Malik RecognizingAction at a Distance InProc. of International Conference onComputer Vision, 2003.

[14] R. Cutler and L. Davis Robust real-time periodic motiondetection, analysis, and applications. InIEEE Trans. PAMI,22(8), August 2000.

[15] L. Zelnik-Manor and M. Irani Event-based video analysis.In Proc of CVPR, 2001.

[16] H. Sidenbladh, M. J. Black and L. Sigal. Implicit proba-bilistic models of human motion for synthesis and tracking.In Proc. European Conference on Computer Vision, vol 1, pp784-800, 2002

[17] M. E. Brand and A. Hertzmann. Style Machines. ACM SIG-GRAPH, pps 183-192, July 2000

[18] B. North and A. Blake and M. Isard and J. Rittscher. Learn-ing and classification of complex dynamics. InIEEE Trans-action on Pattern Analysis and Machine Intelligence, volume22(9), pages 1016-34, 2000.

[19] M. J. Black and A. D. Jepson. A Probabilistic framework formatching temporal trajectories: Condensation-based recogni-tion of gestures and expressions. InProc. of European Con-ference on Computer Vision, volume 1, pages 909-24, 1998.

[20] R. Polana and R. C. Nelson Detection and recognition of pe-riodic, non-rigid motion. InInt. Journal of Computer Vision,23(3):261-282, 1997.

[21] H. A. Rowley and J. M. Rehg Analyzing articulated motionusing expectation-maximization. InProc. CVPR, 1997

[22] M. K. Hu Visual pattern recognition by moment invariants.In IRE Trans. Inf. Theory, IT-8, 179-187, 1962

6

[23] J. Flusser and T. Suk Pattern Recognition by Affine MomentInvariants InPattern Recognition, vol 26, No. 1, pp. 167-174,1993

[24] V. N. Vapnik The Nature of Statistical Learning Theory.Springer, 1995.

[25] T. Joachims Making large-Scale SVM Learning Practical.Advances in Kernel Methods - Support Vector Learning, B.Schoelkopf and C. Burges and A. Smola (ed.), MIT-Press,1999.

[26] K. De Coch and B. De Moor. Subspace angles and distancesbetween arma models.Proc. of the Intl. Symp. of Math. The-ory of Networks and Systems, 2000.

[27] R. Martin. A metric for arma processes.IEEE Trans. onSignal Processing, 48(4):1164–1170, 2000.

[28] L. Ljung. System Identification: theory for the user. PrenticeHall, 1987.

[29] T. Kailath. Linear Systems. Prentice Hall, 1980.

[30] K Arun and S. Y. Kung. Balanced approximation of stochas-tic systems.SIAM Journal of Matrix Analysis and Applica-tions, 11(1):42–68, 1990.

[31] P. Van Overschee and B. De Moor. Subspace algorithms forthe stochastic identification problem.Automatica, 29:649–660, 1993.

[32] A. Weinstein Almost invariant submanifolds for compactgroup actions Berkeley CPAM Preprint Series n.768, 1999

[33] P. V. Overschee and B. De Moor N4SID: Subspace Al-gorithms for the Identification of Combined Deterministic-Stochastic Systems. InAutomatica, Special Issue on Statis-tical Signal Processing and Control, Vol. 30, No.1, 1994, pp.75-93

[34] G. Johansson. Visual perception of biological motion anda model for its analysis. Perception and Psychophysics,14(2):201–211, 1973.

7