1

Multiple Object Tracking Using K-Shortest Paths

OptimizationPATTERN ANALYSIS AND MACHINE INTELLIGENCE,

VOL. 33, NO. 9, SEPTEMBER 2011

2

OUTLINE

• INTRODUCTION• ALGORITHM• RESULTS• CONCLUSION

3

OUTLINE

• INTRODUCTION• ALGORITHM• RESULTS• CONCLUSION

4

INTRODUCTION

• MULTI-OBJECT tracking can be decomposed into two separate steps :1) time-independent detection

generative modelmachine learning-based algorithm

2) link detections into the most likely trajectories

• While it is easy to design a statistical trajectory model with all of the necessary properties for good filtering

NP-complete

5

INTRODUCTION

• This has been dealt with in the literature :1) sampling and particle filtering2) linking short tracks generated using Kalman filtering3) greedy Dynamic Programming• None of these approaches guarantees a global optimum

6

INTRODUCTION

• A notable exception is :A Linear Programming Approach for Multiple Object Tracking[4]

1) priori specifying the number of objects being tracked2) the detector produces false negatives

• Reformulating the linking step• Constrained flow optimization results in a convex problem that fits

into a standard Linear Programming framework• However, yields a very large system that is hardly tractable

7

INTRODUCTION

• Due to its particular structure, the k-shortest paths algorithm, which yields real-time performance on realistically sized problems• Our method does not present any of the limitations mentioned

above, nor does it require an appearance model (optional)• Moreover, it is far simpler both formally and algorithmically than

existing techniques

8

INTRODUCTION

• It performs well in two difficult real-world scenarios :1) tracking multiple balls of similar color, which is a case where an

appearance model would not help2) tracking multiple people with multiple cameras set at shoulder

level so that there are significant occlusions• In both cases, we use an object detector that produces a probabilistic

occupancy map (POM)

9

OUTLINE

• INTRODUCTION• ALGORITHM• RESULTS• CONCLUSION

10

ALGORITHM

• Formulate multitarget tracking as an Integer Programming (IP) problem• A relaxation of it as a Linear Program in polynomial time• However, the large number of variables and constraints makes it

tractable only for small areas and short sequences• K-shortest paths algorithm

NP-hard in many cases

11

Formalization

12

Formalization

incoming flows?

13

Formalization

• In general, some objects may appear inside the tracking area and others may leave• Thus, the total mass of the system changes and we must allow flows

to enter and exit the area

14

Formalization

virtual locations

15

Formalization

Our goal :

feasible maps

satisfies (1), (2), (3), and (4)

16

Formalization

• Assuming conditional independence of the , given the , the optimization problem of (6) can be rewritten as :

17

Linear Programming Formulation

• The formulation defined above translates naturally into the Integer Program :

Why?

a≥b≥…≥c≥d

a≤d

a=b=…=c=d

18

Linear Programming Formulation

• This new formulation is strictly equivalent to the original one and no additional constraint is needed• The inequalities are indeed sufficient to ensure that no flow can ever

appear or disappear within the graph• Under this formulation, our Integer Program can be solved by any

generic LP solver NP-completePolynomial

Total Unimodularity

19

Unimodular Matrix

• A unimodular matrix M is a square integer matrix having determinant +1 or −1• Equivalently, it is an integer matrix that is invertible over the integers• Thus every equation Mx = b, where b is integer, and M is unimodular,

has an integer solution x = b

20

Total Unimodularity

• A totally unimodular matrix (TU matrix) is a matrix for which every square non-singular(or called invertible) submatrix is unimodular• From the definition it follows that any totally unimodular matrix has

only 0, +1 or −1 entries (The opposite is not true)• If A is totally unimodular and b is integral, every extreme point of the

feasible region is integral and thus the feasible region is an integral polyhedron

21

Linear Programming Formulation

• Generic LP solvers : Simplex algorithm [5] Interior-point-based methods [36]

• However, this approach would only be tractable for moderately sized problems and does not scale to most practical applications• And have very high worst-case time complexities

22

K-Shortest Paths Formulation

• Given a pair of nodes, namely, the source and the sink , in a graph G, the k-shortest paths problem is to find the k paths {, } between these nodes such that the total cost of the paths is minimum• In our specific case, we are interested in :

1) the graph is directed2) paths are both node-disjoint and node-simple

3) only looking for paths between the source and sink nodesdirected acyclic graph (DAG)

23

K-Shortest Paths Formulation

• The cost value of the edges emanating from the source node is set to zero

feasible solutions of the originalLP formulation of (11)

24

K-Shortest Paths Formulation

• Let be the shortest path computed at the th iteration of the algorithm• = {,…, } be the set of all shortest paths computed up to iteration

• The optimal number of paths is obtained when the cost of iteration is higher than the one of iteration

25

K-Shortest Paths Formulation

• To compute such k-shortest paths, we use the disjoint paths algorithm [6]• The worst-case complexity of the algorithm is • Furthermore, due to the mostly acyclic nature of our graph, the

average complexity is almost linear with the number of nodes

26

Batch Processing and Complexity Reduction• Processing a whole video sequence is possible but impractical for

applications such as broadcasting, in which the result must be supplied quickly• We split the sequence into batches of 100 frames• This results in a constant 4 second delay between input and output,

which is nevertheless compatible with many applications• To enforce temporal consistency across batches, we add the last

frame of the previously optimized batch to the current one

27

Batch Processing and Complexity Reduction• Since most of the probabilities of presence estimated by the detector

are virtually equal to zero• We can reduce the number of nodes

• In the examples presented in this paper, we have not found it necessary to do so

28

OUTLINE

• INTRODUCTION• ALGORITHM• RESULTS• CONCLUSION

29

RESULTS

• First, we use a multicamera setup in which the cameras are located at shoulder level to track pedestrians who may walk in front of each other• As a result, our approach was shown to compare favorably against

other state-of-the-art algorithms in the PETS 2009 evaluation [37]• Second, to highlight the fact that we do not depend on an appearance

model, we track sets of similar-looking bouncing balls seen from above• Compare to sequential Dynamic Programming and show that we can

obtain good results even when using a single camera

30

Probabilistic Occupancy Map

• We used the publicly available implementation [40] of our earlier POM algorithm [3] to create the detection data needed as input by our tracker• This method performs binary background/foreground segmentation

and then uses a generative model to estimate the most likely locations• The generative model at the heart of POM represents people as

cylinders that project to rectangles in the images

[40] POM: Probabilistic Occupancy Map

31

Probabilistic Occupancy Map

• In our model, the resolution of the ground grid is independent of the target’s size• If grid cells are smaller than a target, the detections do not spread

over several cells• POM implicitly performs a nonmaximum suppression (peaky)

32

Evaluation Metrics

• Video Analysis and Content Extraction (VACE) programSequence Frame Detection Accuracy (SFDA)Average Tracking Accuracy (ATA)SODA

• Classification of Events, Activities, and Relationships (CLEAR) consortiumMultiple Object Detection Accuracy (MODA)Multiple Object Detection Precision (MODP)Multiple Object Tracking Accuracy (MOTA)Multiple Object Tracking Precision (MOTP)

33

Evaluation Metrics

• MODP : the quality of the bounding box alignment in case of correct detection• MODA : false positives and missed detections• MOTP : the alignment of tracks with the ground truth• MOTA : false positives, missed detections, and identity switches

34

35

Test Data

• Laboratory Sequence• Basketball Sequence• Passageway Sequences• PETS 2009 Sequence

36

37

Test Data

• Monocular Pedestrian

38

Test Data

• Ball Tracking

39

Evaluation Metrics

• To quantify our results, we manually labeled some of the test sequences :

Sequences Frames Frames per Labeled

Ball * 2 1,000 3

PETS 2009 sequence S2/L1 800 5

Passageway * 4 2,500, 800, 900, and 800 25

Laboratory 5,000 25

40

41

42

43

44

Failure Modes

• Our tracking algorithm can be mainly affected by two elements : false detections missing ones

• Selected a 1,000 frame excerpt of the laboratory sequence1) added various levels of random detection noise uniformly2) randomly deleted detections from the same original sequence

45

46

47

= = 3

48200-frame excerptof the laboratory sequence

linear

49

OUTLINE

• INTRODUCTION• ALGORITHM• RESULTS• CONCLUSION

50

CONCLUSION

• Combining frame-by-frame detections to estimate the most likely trajectories of an unknown number of targets, including their entrances and departures to and from the scene, is one of the most difficult components of a multiobject tracking algorithm• Formalizing the motions of targets as flows• Standard Linear Programming• K-shortest paths algorithm• Performing robust multi-object tracking in real time

51

CONCLUSION

• The resulting algorithm is far simpler than current state-of-the-art alternatives• Ensures that a global optimum can be found• Future work will focus on integrating additional cues to our

framework, such as an appearance or a motion models, to robustly handle identities of intersecting trajectories

52

Thanks for Listening!