II INTERNATIONAL CONFERENCE ON COMPUTATIONAL BIOENGINEERING H. Rodrigues et al. (Eds.) Lisbon, Portugal, September 14-16, 2005 HUMAN MOVEMENT TRACKING AND ANALYSIS WITH KALMAN FILTERING AND GLOBAL OPTIMIZATION TECHNIQUES Raquel R. Pinho*, João Manuel R. S. Tavares* and Miguel V. Correia** * Instituto de Engenharia Mecânica e Gestão Industrial, Lab. de Óptica e Mecânica Experimental Universidade do Porto, Faculdade de Engenharia, Dept. Eng. Mecânica e Gestão Industrial e-mail: [email protected], [email protected]** Instituto de Engenharia Biomédica, Lab. Sinal e Imagem Universidade do Porto, Faculdade de Engenharia, Dept. Eng. Electrotécnica e de Computadores e-mail: [email protected]Keywords: Kalman filter, Optimization, Mahalanobis distance, Movement analysis, Tracking. Abstract. This paper addresses the problem of tracking feature points along image sequences to analyze the undergoing human movement. An approach based on Kalman filtering performs the estimation and correction of the feature point’s movement in every image frame, and optimizes the incorporation of the measured data in order to establish the best global correspondence. We propose a criterion to establish correspondences between the group of estimates in each image and the new data to include, using an optimization criterion that minimizes the global matching cost based on the Mahalanobis distance. With this solution, we are also able to deal with the occlusion and appearance of feature points along with the movement tracking. In this paper we present experimental results obtained on real human tracking images that validate our approach.
Transcript
II INTERNATIONAL CONFERENCE ON COMPUTATIONAL BIOENGINEERING H. Rodrigues et al. (Eds.)
Lisbon, Portugal, September 14-16, 2005
HUMAN MOVEMENT TRACKING AND ANALYSIS WITH KALMAN FILTERING AND GLOBAL OPTIMIZATION TECHNIQUES
Raquel R. Pinho*, João Manuel R. S. Tavares* and Miguel V. Correia**
* Instituto de Engenharia Mecânica e Gestão Industrial, Lab. de Óptica e Mecânica Experimental Universidade do Porto, Faculdade de Engenharia, Dept. Eng. Mecânica e Gestão Industrial
Keywords: Kalman filter, Optimization, Mahalanobis distance, Movement analysis, Tracking.
Abstract. This paper addresses the problem of tracking feature points along image sequences to analyze the undergoing human movement. An approach based on Kalman filtering performs the estimation and correction of the feature point’s movement in every image frame, and optimizes the incorporation of the measured data in order to establish the best global correspondence. We propose a criterion to establish correspondences between the group of estimates in each image and the new data to include, using an optimization criterion that minimizes the global matching cost based on the Mahalanobis distance. With this solution, we are also able to deal with the occlusion and appearance of feature points along with the movement tracking. In this paper we present experimental results obtained on real human tracking images that validate our approach.
Raquel R. Pinho, João M. R. S. Tavares and Miguel V. Correia
1 INTRODUCTION
Human movement analysis has, for many years, been the centre of numerous research studies in several fields, for example medical diagnosis, physical therapy or sports. These applications can be used, e.g., to study gait disorders related to knee or hip pain or tumors, to control cycles of motion for rehabilitation or training.
Gait velocity, stride length, stride rate, angular velocity, accelerations, velocities and displacements of different body parts, and other related joint angle parameters are the variables most often examined when studying human movement patterns. However, the complexity of the interactions of the various components of human movement have stimulated some technologies such as force platforms, electromyographic systems, high-speed cameras and computerized movement analysis laboratories, which have allowed new insights into human bipedal ambulation locomotion [1].
The experienced professionals may perform functional gait assessment in the absence of most of the technological assistance [1]. But human movement analysis with motion capture video systems and interactive modelling systems can help the analysis, diagnosis, and documentation of movements with tools that may, for example, compare the gaits of patients before and after treatment to historical case studies of normal and abnormal gaits. Our aim is to develop a system which can automatically track human movement, and that can subsequently be used to classify human movement.
To track the human movement we used some widespread tracking techniques based on markers and a stochastic method, the Kalman filter, but combined them with optimization techniques in order to increment these methods robustness to occlusion and non-linear movement.
Although some problems have raise up towards the use of markers such as the difference between tracking the skin surface instead of the joint itself [2], it still is a widely used technique especially due to its robustness to occlusion during relatively long time intervals [3, 4].
When tracking the movement of body parts, we used difference equations to model the markers trajectories with time, and the measurements were obtained at discrete instances at every image frame. In doing so, a stochastic method was used, as the involved parameters cannot be controlled nor calculated deterministically: the underlying kinematics model is not perfect (approximations must be done), and sensors don’t provide perfect and complete data about the system. So, we use a well known Bayesian procedure, the Kalman filter. The Kalman filter estimates a process by using a form of feedback control: the filter estimates the process state at some time and then obtains feedback in the form of (noisy) measurements. The drawbacks of this method are related to its relatively high restrictive assumptions [5]. In this paper we prove the use of optimization techniques and Mahalanobis distance to reduce some of the effects of the Kalman’s filter when those restrictions are not satisfied, which is the case of tracking human movement.
This paper is organized as follows. In the next section a brief introduction is made to the Kalman filter. In section 3, we describe our solution to the incorporation of new data into the filter by using optimization techniques with Mahalanobis distance. Then some experimental results are shown on real human movement images. In the last section some conclusions will
Raquel R. Pinho, João M. R. S. Tavares and Miguel V. Correia
be held and perspectives on future work will be given.
2 THE KALMAN FILTER
The Kalman filter is an optimal recursive Bayesian stochastic method. It provides “optimal” estimates that minimize the mean of squared error. In a Bayesian stochastic viewpoint the filter propagates conditional probability density of the system state conditioned on the knowledge of the actual data coming from the measuring devices. In this work, the system state in each time step is the set of positions, velocities and accelerations of the used markers, and new measurements are incorporated whenever a new image frame is considered.
The equations for the Kalman filter fall into two groups: time update (or prediction) equations and measurement update (or correction) equations. The time update equations are responsible for projecting forward (in time) the current state and error covariance estimates to obtain the a priori estimates for the next time step. The measurement update equations are responsible for the feedback—i.e. for incorporating a new measurement into the a priori estimate to obtain an improved a posteriori estimate [6].
The prediction step is based on the Chapman-Kolmogorov equation for a first order Markov process:
1t tx x− +−= Φ , (1)
where relates the system state Φ 1tx+− at the previous time step 1t − to the state tx− at the
current step t . The superscripts + and − indicate if measurement data have been or not respectively incorporated. The related uncertainty is given by:
1T
t tP P− +− Q= Φ Φ + , (2)
where P is the covariance matrix and models the process noise. QThe correction equations that update the predicted estimates upon the incorporation of new measurements are given by:
tu
1T T
t t t tK P H HP H R−− −⎡ ⎤= +⎣ ⎦ , (3)
t t t t tx x K u Hx+ − −⎡ ⎤= + −⎣ ⎦ , (4)
[ ]t tP I K H Pt+ −= − , (5)
where K is chosen to be the gain that minimizes the a posteriori error covariance equation, H processes the coordinates transformation between the predicted space and the measurement space, is the measurement noise involved, and is the identity matrix [6, 5]. tR IOne of the drawbacks of the Kalman filter is the restrictive assumption of Gaussian posterior density functions at every time step and many tracking problems involve non-linear movement, human gait is just an example. To solve this problem, several techniques have been proposed
Raquel R. Pinho, João M. R. S. Tavares and Miguel V. Correia
such as the Extended Kalman Filter, the Unscented Kalman Filter or other stochastic methods such as particle filters [5, 7, 8], but their computational expense may be questioned.
3 INCORPORATION OF NEW DATA WITH OPTIMIZATION TECHNIQUES AND MAHALANOBIS DISTANCE
To introduce new measurement data for the correction equations there must exist some criterion of correspondence between the set of measurements and the given estimates, so that for each markers position estimate there may exist at most one new measurement to correct its predicted position.
By Kalman’s default approach the search area for each marker position in the image plane is given by an ellipse whose axes are determined by the eigenvectors of the covariance reduced matrix, its rays are given by the associated eigenvalues, and centred in the previously predicted position [9, 10]. If the filter converges, better estimates will be given and the search areas will successively decrease to a minimum value, so the computational cost due to the points correspondence decreases [9, 10].
However this default approach may present some problems: there may not exist any point in the search area, or there might be several points in the same search area. And even if there is only one correspondence for each point, there is no guarantee if the best set of correspondences has been achieved. Next, we present an approach that might surpass such ambiguities: we propose the use of optimization techniques, in order to obtain the best set of correspondences between the previous predictions and the actual set of measurements. We consider the whole set of measurements and with a global optimization method, e.g. the Simplex method, to establish the best global set of correspondences with the given estimates. The cost of each correspondence is given by the normalized Mahalanobis distance [10].
3.1 The Simplex method in the measurement update
The Simplex algorithm is an iterative algebraic procedure that can be used to determine at least one optimal solution for each problem [11, 12].
As a linear optimization method the Simplex algorithm optimizes a function which is subject to some restrictions. In the case of tracking markers, we wish to minimize the global cost of correspondences between the set of measurements and estimates of the markers positioning. To do so it should be noticed that for each estimate there will be given at most one measurement, and that each new measurement will be associated to a marker position. So, we use the assignment formulation of the Simplex algorithm [11].
3.2 Mahalanobis distance
To optimize the correspondences between the set of measurements and the set of given estimates, we had to associate a cost to each correspondence. In this work, we used the Mahalanobis distance to determine the cost of each correspondence.
The Mahalanobis distance between two points is scaled by the statistical variation in each component of the point. So if EX is the markers position estimate, and MX is a measurement,
Raquel R. Pinho, João M. R. S. Tavares and Miguel V. Correia
then their Mahalanobis distance is given by:
2
1( ) ( ) (2s
TM E M E M E )X X V V X X
dχ
−− + −= , (1)
where MV is the variance of each position vector of the measurement, and EV is associated to the points prediction [10].
In our work, we considered the variance of each estimate given by the Kalman’s filter, so it is the area of the ellipse related to the variance of the prediction error. However, as the variance of each measurement is not defined, we considered it the area of the smallest ellipse that contains the measurement and that has the same centre, axes and maintains the axes proportion of the prediction ellipse as figure 1.
XE
XM
Figure 1: Variance of each measurement MX given by the dashed ellipse area
if the variance of EX is the area of the solid ellipse.
The Mahalanobis distance values will be inversely proportional to the quality of the prediction/measurement correspondence, so to optimize correspondences we minimize this cost function.
3.3 Marker occlusion or appearance during movement tracking
One of the previously mentioned restrictions on the assignment formulation of the Simplex algorithm is the correspondence between each measurement and a Kalman’s prediction. However, when markers are occluded or appear in the tracking sequence that restriction does not stand. To overcome this difficulty, we add fictitious variables so that the number of tracked markers equals the number of given measurements. The cost due to each correspondence with a fictitious variable is null. So, when the Simplex finds the optimal set of correspondences, all points that match with a fictitious variable are considered unmatched [11].
If a predicted marker position does not get any correspondent in the measurement set it is assumed that the marker has been occluded, but the tracking of that point is maintained by including its predicted position in the measurement vector, although with higher uncertainty. For this paper we squared the values of the predicted error covariance (note that these values are greater that 1 so this will increase the associated variance).
Raquel R. Pinho, João M. R. S. Tavares and Miguel V. Correia
If a measurement does not correspond to any of the marker predicted position then we consider it a new marker and initialize its tracking.
4 EXPERIMENTAL RESULTS
In our previous work we have exeHere, we will concentrate on its applica
mplified the above approach upon synthetic data [13]. tion to real human movement sequences.
e rate of 200 fra
dot, which in the image plane cor
Consider a sequence of 7 frames where 6 markers have been placed on a walking man’s leg (figure 2). This image sequence was captured by a high speed camera with a fram
mes/second, and the obtained images are 256x256 pixels2. All markers are alike and are composed by a white rectangle that contains a black dot. With
the Kalman filter we track the centre of the markers blackresponds to the centre of a blob of approximately 3x3 pixels2. In the first frame only 5
markers (markers numbered from 1 to 5 in figure 2) are visible, but in the next five frames they are all visible. In the last frame a marker is occluded but its tracking should continue, as suggested in figure 3, but with higher uncertainty.
1
1
2 3
4
5
2
Figure 2: Tracking 6 markers in a human moving leg: 1 - original 1st. frame; 2 – 1st. frame with measurements signed with x and 5 visible markers numbered.
Raquel R. Pinho, João M. R. S. Tavares and Miguel V. Correia
1
2
3
4
5
6
Figure 3: Tracking 6 markers in a 7 frame image sequence: Kalman’s search area defined by solid ellipses, the predicted position for each marker is given by +,
and the corrected position is represented with x.
Raquel R. Pinho, João M. R. S. Tavares and Miguel V. Correia
Analysing the numerical results obtained by Kalman’s filter for the given example we notice that although the tracked movement is non-linear, the error between the filters estimates and the introduced measure is always less than 3.35 pixels (obtained at marker 2 when ), but the average error on each marker position is of 1.2 pixels (table 1 and 2). Plus, with time the filter tends to get better results (for example in for
2t =
5t = all estimates are less than 1 pixel from the correct measure), but as the human movement is non-linear this is not a general result (as in the next step, 6t = , 2 markers overcome the 1 pixel distance).
Please note that the filter we used can also predict and correct the velocity and acceleration of the tracked markers, but we have omitted these results in sake of simplicity and briefness.
Table 2. Some of the numerical results of Kalman’s filter in figure 3.
Analysing the Kalman’s defined search area for each marker we notice, as expected, that although starting with areas of 452.4 pixels2 the ellipses areas tend to decrease to a minimal value around 28.3 pixels2 which occurs to all markers in steps 4 and 5 of figure 3. If a marker disappears in some sequence image, than its uncertainty will increase, this is the case of marker 1 in step 6, whose uncertainty grows to 226.2 pixels2. On the other hand, when a marker appears it is initialized with the default values so its behaviour is unlike alike the others.
For the next example 5 markers were placed on a man’s lower leg (figure 4). The used markers are similar to the ones in the previous example, and the captured images have the same dimensions. We used 5 non-consecutive frames taken from a video sequence with intervals of 10 frames (we choose these intervals in order to increase the displacement between steps of the Kalman filter).
Analysing the tracking results represented in figure 5, we notice that although there may not exist any new measurement in the search area of each markers position, our approach using global optimization and the Mahalanobis distance, can correctly introduce new data. This result demonstrates the robustness of our approach when tracking non-linear movement.
Raquel R. Pinho, João M. R. S. Tavares and Miguel V. Correia
1 2
Figure 4. Tracking 5 markers in a human moving lower leg: 1 - original 1st. frame; 2 – 1st. frame with signed measurements.
5 CONCLUSIONS AND FUTURE WORK
In this paper we have addressed the problem of tracking feature points along image sequences to analyse human movement. To do so, we used the Kalman filter which is able to predict and correct the tracked points position, as well as their velocity and acceleration. To incorporate correctly the data contained in each image frame we used optimization techniques and Mahalanobis distance. This approach also allows the incorporation of data even if it would be out of the default Kalman search area, which happens for example in the case of a change of direction of movement.
This work should be continued with the application of this approach to human movement sequences without any markers. The obtained results should also be compared to those given by other stochastic methods used for tracking such as the unscented Kalman filter, or particle filters. This work can also be enriched through the analysis of the data obtained by human movement experts for example to diagnose pathological human gait.
ACKNOWLEDGMENTS
The first author would like to thank the support of the PhD grant SFRH / BD / 12834 / 2003 of the FCT - Fundação de Ciência e Tecnologia in Portugal.
Raquel R. Pinho, João M. R. S. Tavares and Miguel V. Correia
This work was partially done in the scope of the project “Segmentation, Tracking and Motion Analysis of Deformable (2D/3D) Objects using Physical Principles”, reference POSC/EEA-SRI/55386/2004, financially supported by FCT in Portugal.
1 2
3 4
Figure 5: Tracking 5 markers in a 5 non-consecutive frame image sequence shows that our approach is robust although the Kalman’s search area does not contain the new data.
Raquel R. Pinho, João M. R. S. Tavares and Miguel V. Correia
REFERENCES
[1] D. Fish, J. Nielsen, Clinical Assessment of Human Gait, Prosthetics and Orthotics Journal, vol. 5, pp. 39/48, 1993.
[2] D. Meyer, J. Denzler, H. Niemann, Model Based Extraction of Articulated Objects in Image Sequences, 4th. International Conference on Image Processing, 1997.
[3] H. Lakany, G. Hayes, M. Hazlewood, S. Hillman, Human Walking: Tracking and Analysis, IEEE Electronic & Communications, Colloquium on Motion Analysis and Tracking, 1999.
[4] W. Park, D. Chaffin, B. Martin, Toward Memory-Based Human Motion Simulation: Development and Validation of a Motion Modification Algorithm, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, vol. 34, 2004.
[5] M. Arulampalam, S. Maskell, N. Gordon, T. Clapp, A Tutorial on Particle Filters for Online Nonlinear/Non-Gaussian Bayesian Tracking, IEEE Transactions on Signal Processing, vol. 50, pp. 174/188, 2002.
[6] G. Welch, G. Bishop, An Introduction to Kalman Filter, University of North Carolina at Chapel Hill, 1995.
[7] C. Shen, A. Hengel, A. Dick, M. Brooks, Enhanced Importance Sampling: Unscented Auxiliary Particle Filtering for Visual Tracking, AI 2004: Advances in Artificial Intelligence, vol. 3339, pp. 180-191, 2004.
[8] A. Sitz, U. Schwarz, J. Kurths, The Unscented Kalman Filter, a Powerful Tool for Data Analysis, International Journal of Bifurcation and Chaos, vol. 14, pp. 2093-2105, 2004.
[9] M. Correia, Análise De Movimento Em Sequências De Imagens, Faculdade de Engenharia, Universidade do Porto, 1995.
[10] J. Tavares, Obtenção De Estrutura Tridimensional a Partir De Movimento De Câmara, Faculdade de Engenharia, Universidade do Porto, 1995.
[11] F. Hillier, G. Lieberman, Introduction to Operations Research. McGraw-Hill International Editions, 2001.
[12] W. Press, S. Teukolsky, B. Flannery, W. Vetterling, Numerical Recipes in C++: The Art of Scientific Computing. Cambridge University Press, 2002.
[13] R. Pinho, J. Tavares, M. Correia, Seguimento De Objectos Em Visão Computacional Usando Métodos Estocásticos, Congreso de Métodos Numéricos en Ingeniería 2005, Granada, Spain, 2005.
