Date post: | 27-May-2015 |
Category: |
Technology |
Upload: | bouwmans-thierry |
View: | 452 times |
Download: | 4 times |
Background Modeling and Foreground Detection for Video Surveillance:
Recent Advances and Future Directions
Thierry BOUWMANSAssociate Professor
MIA Lab - University of La Rochelle - France
2
Plan
Introduction Fuzzy Background Subtraction Background Subtraction via a
Discriminative Subspace Learning: IMMC Foreground Detection via Robust Principal
Component Analysis (RPCA) Conclusion - Perspectives
3
Goal
Detection of moving objects in video sequence.
Pixels are classified as: Foreground (F)Background(B)
Séquence Pets 2006 : Image298 (720 x 576 pixels)
4
Background Subtraction Process
Background Maintenanc
e
ForegroundDetection
Background Initialization
F(t)Foreground Mask
Video
t ≤ N
t > N
t ≥ N t=t+1
N images
I(t+1)N+1
Incremental Algorithm
Classification task
Batch Algorithm
5
Related Applications
Video surveillance Optical Motion Capture Multimedia Applications
Projet ATON – Université de Californie San Diego
Séquence Danse [Mikic 2002] Séquence Jump [Mikic 2002]Projet Aqu@theque – Université de La Rochelle
6
Background Subtraction
Processing
Acquisition
Convex Hull
Pattern Recognition
Tracking
On the importance of the background subtraction
7
Challenges
Critical situations which generate false detections :
Shadows -
Illumination variations…
Source : Séquence Pets 2006 Image 0298 (720 x 576 pixels)
Multimodal Backgrounds
Rippling Water
Water Surface
Camera Jitter
Waving Trees
8
Source: http://perception.i2r.a-star.edu.sg/bk_model/bk_index.html
9
Background Subtraction Web Site: References (553), datasets (10) and codes (27).
Statistical Background Modeling
Source: http://sites.google.com/site/backgroundsubtraction/Home.html (6256 Visitors, Source Google Analytics).
10
Plan
Introduction Fuzzy Background Subtraction Background Subtraction via a
Discriminative Subspace Learning: IMMC Foreground Detection via Robust Principal
Component Analysis (RPCA) Conclusion - Perspectives
11
Fuzzy Background Subtraction
A survey in Handbook on Soft Computing for Video Surveillance, Taylor and Francis Group [HSCVS 2012]
Three approaches developed at the MIA Lab: Background modeling by Type-2 Fuzzy Mixture of
Gaussians Model [ISVC 2008]. Foreground Detection using the Choquet Integral
[WIAMIS 2008][FUZZ’IEEE 2008] Fuzzy Background Maintenance [ICIP 2008]
12
Weakness of the original MOG1. False detections due to the matching test
13
Weakness of the original MOG2. False detections due to the presence of outliers in the training
step
Exact distribution
14
Mixture of Gaussians with uncertainty on : the mean and the variance [Zeng 2006]
(T2 FMOG-UM) (T2 FMOG-UV)
15
Mixture of Gaussians with uncertainty on the mean(T2 FMOG-UM)
: Intensity vector in the RGB color space
16
Mixture of Gaussians with uncertainty on the variance (T2 FMOG-UV)
: Intensity vector in the RGB color space
17
Classification B/F by T2-FMOG
Matching test:
Classification B/F as the MOG ⇒
Results on the “SHAH” dataset(160 x 128 pixels) – Camera Jitter
Original sequence MOG
T2 FMOG-UM (km=2) T2 FMOG-UV (kv=0.9)18
Video at http://sites.google.com/site/t2fmog/
Results on the “SHAH” dataset(160 x 128 pixels) – Camera Jitter
Method Error Type
Image 271
Image 373
Image 410
Image 465
Total Error
Variation in %
MOG FNFP
02093
11204124
48182782
20501589 18576
T2-FMOG-UM FNFP
0203
1414153
6043252
252046 10631 42,77
T2-FMOG-UV FNFP
03069
9571081
22171119
10691158 10670 42.56
19
Results on the “SHAH” dataset(160 x 128 pixels) – Camera Jitter
20
[Stauffer 1999]
[Bowden 2001] – Initialization [Zivkovic 2004] – K is variable
Results on the sequence “CAMPUS” (160 x 128 pixels) – Waving Trees
Original Sequence MOG
T2 FMOG-UM (km=2) T2 FMOG-UV (kv=0.9)21
Video at http://sites.google.com/site/t2fmog/
Resultat on the sequence “Water Surface” (160 x 128 pixels) – Water Surface
Original Sequence MOG
T2 FMOG-UM (km=2) T2 FMOG-UV (kv=0.9)22
Video at http://sites.google.com/site/t2fmog/
23
Fuzzy Foreground Detection :
Features: color, edge, stereo features, motion features, texture.
Multiple features: More robustness in presence of illumination
changes, shadows and multimodal backgrounds
24
Choice of the features
Color (3 components)
Texture (Local Binary Pattern [Heikkila – PAMI
2006])
For each feature, a similarity (S) is computed following its value in the background image and its value in the current image.
25
Aggregation of the Color and Texture features with the Choquet Integral
Color Features Texture Features
Similarity mesure for the
Color
Similarity measure for the
Texture
Fuzzy Integral
Classification B/F
Foreground Mask
BG(t)
I(t+1)
SC,1 SC,2 SC,3 ST
How to compute S for the Color and the Texture?
T<TifT
TT=Tif1
T<TifT
T
=S
BIB
I
IB
IBI
B
T
IT
k I,C
Background Image
Current Image
For the TextureFor the Color
kFkIkF
kI
kIkF
kIkFkI
kF
kC
CCifC
CCCif
CCifC
C
S
,,,
,
,,
,,,
,
, 1
FT
k F,C
0 ≤ S ≤ 1
k=one of the color components
0 ≤ T,C ≤ 255
26
27
Fuzzy operators
« Sugeno Integral» et «Choquet Integral»
Uncertainty and imprecision Great flexibility Fast and simple operations
ordinal cardinal
28
Data Fusion using the Choquet Integral
Mesures floues :
Intégrale de Choquet :
29
Fuzzy Foreground Detection
Classification using the Choquet integral
If then else
where Th is constant threshold. is the value of the Choquet integral for the pixel (x,y)
Aggregation Color, Texture Aqu@thèque (384 x 288 pixels) - Ohta color space
a) Current image b) Ground truth
c) Choquet integral d) Sugeno integral [Zhang 2006]
IntegralColor space
ChoquetOhta
SugenoOhta
S(A,B) 0.40 0.27
Comparison between the Sugeno and Choquet Integrals30
Aggregation Colors, Texture : Ohta, YCrCb, HSV Aqu@thèque (384 x 288 pixels)
Choquet - Ohta Choquet - YCrCb Choquet - HSV
IntegralColor Space Ohta YCrCb HSV
S(A,B) 0.40 0.42 0.30Evaluation of the Choquet integral for different color spaces
0.6
0.5
0.5
0.5
0.53
0.3
0.4
0.3
0.39
0.34
0.1
0.1
0.2
0.11
0.13
0.9
0.9
0.8
0.89
0.87
0.7
0.6
0.7
0.61
0.66
0.4
0.5
0.5
0.5
0.47
1
1
1
1
1
Values of the fuzzy measures
31
Texture
Color
32
Aggregation Color, Texture VS-Pets 2003 (720 x 576)
Current Image Choquet - YCrCb Sugeno – Ohta [Zhang 2006]
Aggregation Colors : Pets 2006 (384 x 288
pixels) Original sequence Ground truth
OR Sugeno Integral Choquet Integral
YCrCb
Ohta
HSV33
34
Fuzzy Background maintenance No-selective rule
Selective rule
Here, the idea is to adapt very quickly a pixel classified asbackground and very slowly a pixel classified as foreground.
35
Fuzzy adaptive rule
Combination of the update rules of the selective scheme
and
Original Image 1850 Ground Truth
No selective rule Selective rule Fuzzy adaptive rule
No selective Selective
Fuzzy adaptive
S(A,B)% 58.40 57.08 58.96
36
Results on the Wallflower datasetSequence Time of Day
Similarity measure
37
Computation Time
Algorithm Frames/Second
T2-FMOG-UM 11
T2-FMOG-UV 12
MOG 20
Choquet integral 31
Sugeno integral 22
OR 40
Resolution 384*288, RGB, Pentium 1,66GHz, RAM 1GB
38
Assessment
Fuzzy Background Modeling by T2-FMOG Multimodal Backgrounds
Fuzzy Foreground Detection using multi-features
Fuzzy Background Maintenance
Perspectives
- Using fuzzy approaches in other statistical models.
- Using more than two features- Fuzzy measures by learning
39
Plan
Introduction Fuzzy Background Subtraction Background Subtraction via a
Discriminative Subspace Learning: IMMC Foreground Detection via Robust Principal
Component Analysis (RPCA) Conclusion - Perspectives
40
Background Modeling and Foreground Detection via a Discriminative Subspace Learning (MIA Lab)
Reconstructive subspace learning models (PCA, ICA, IRT) [RPCS 2009]
Assumption: The main information contained in the training sequence is the background meaning that the foreground has a low contribution.
However, this assumption is only verified when the moving objects are either small or far away from the camera.
41
Discriminative Subspace Learning
Advantages More efficient and often give better classification results. Robust supervised initialization of the background Incremental update of the eigenvectors and eigenvalues.
Approach developed at the MIA Lab: Background initialization via MMC [MVA 2012] Background maintenance via Incremental Maximum
Margin Criterion (IMMC) [MVA 2012]
42
Background Subtraction via Incremental Maximum Margin Criterion
Denote the training video sequences S ={I1, ...IN}
where It is the frame at time t
N is the number of training frames.
Let each pixel (x,y) be characterized by its intensity in the grey scale and asssume that we have the ground truth corresponding to this training video sequence, i.e we know for each pixel its class label that can be foreground or background.
43
Background Subtraction via Incremental Maximum Margin Criterion
Thus, we compute respectively the inter-class scatter matrix Sb and the intra-class scatter matrix Sw:
where c = 2
I is the mean of the intensity of the pixel (x,y) over the training video
Ii is the mean of samples belonging to class i
pi is the prior probability for a sample belonging to class i (Background,
Foreground).
44
Background Subtraction via Incremental Maximum Margin Criterion
Batch Maximum Margin Criterion algorithm.
Extract the first leading eigenvectors that correspond to the background. The corresponding eigenvalues are contained in the matrix LM and the leading eigenvectors in the matrix ΦM.
The current image It can be approximated by the mean background and weighted sum of the leading eigenbackgrounds ΦM.
45
Background Subtraction via Incremental Maximum Margin Criterion
The coordinates in leading eigenbackground space of the current image It can be computed :
When wt is back projected onto the image space, the background image is created :
46
Background Subtraction via Incremental Maximum Margin Criterion
Foreground detection
Background maintenance via IMMC
47
Principle - Illustration
Current Image
IBackground
IForeground
Background image
Foreground mask
48
Results on the Wallflower dataset
Original image, ground truth , SG, MOG, KDE, PCA, INMF, IRT, IMMC (30), IMMC (100)
49
Assessment
Advantages Robust supervised initialization of the background. Incremental update of the eigenvectors and
eigenvalues.
Disadvantages Needs ground truth in the training step.
Others Discriminative Subspace Learning methods such as LDA.
Perspectives
50
Plan
Introduction Fuzzy Background Subtraction Background Subtraction via a
Discriminative Subspace Learning: IMMC Foreground Detection via Robust Principal
Component Analysis (RPCA) Conclusion - Perspectives
51
Foreground Detection via Robust Principal Component Analysis
PCA (Oliver et al 1999): Not robust to outliers. Robust PCA (Candes et al. 2011):
Decomposition into low-rank and sparse matrices
Approach developed at the MIA Lab: Validation [ICIP 2012][ICIAR 2012][ISVC 2012] RPCA via Iterative Reweighted Least Squares [BMC
2012]
52
Robust Principal Component Analysis
Candes et al. (ACM 2011) proposed a convex optimization to address the robust PCA problem. The observation matrix A is assumed represented as:
where L is a low-rank matrix and S must be sparse matrix with a small fraction of nonzero entries.
http://perception.csl.illinois.edu/matrix-rank/home.html
53
Robust Principal Component Analysis
This research seeks to solve for L with the following optimization problem:
where ||.||* and ||.||1 are the nuclear norm (which is the l1-norm of singular value) and l1-norm, respectively, and λ > 0 is an arbitrary balanced parameter.
Under these minimal assumptions, this approach called Principal Component Pursuit (PCP) solution perfectly recovers the low-rank and the sparse matrices.
54
Algorithms for solving PCP
Algorithms Accuracy Rank ||E||_0 # iterations
time (sec)
IT 5.99e-006 50 101,268 8,550 119,370.3
DUAL 8.65e-006 50 100,024 822 1,855.4
APG 5.85e-006 50 100,347 134 1,468.9
APGP 5.91e-006 50 100,347 134 82.7
ALMP 2.07e-007 50 100,014 34 37.5
ADMP 3.83e-007 50 99,996 23 11.8
10,000timesspeedup!
Time required to solve a 1000x1000=106 RPCA problem:
Source: Z. Lin , Y. Ma “The Pursuit of Low-dimensional Structures in High-dimensional (Visual) Data: Fast and Scalable Algorithms”
Time required is still acceptable for ADM but for background modeling and foreground detection?
55
Application to Background Modeling and Foreground Detection
Source: http://perception.csl.illinois.edu/matrix-rank/home.html
n is the amount of pixels in a frame (106)m is the number of frames considered (200)Computation time is 200* 12s= 40 minutes!!!
56
PCP and its application to Background Modeling and Foreground Detection
Only visual validations are provided!!!
Limitations:
Spatio-temporal aspect: None! Real Time Aspect: PCP takes 40 minutes with the
ADM!!! Incremental Aspect: PCP is a batch algorithm. For
example, (Candes et al. 2011) collected 200 images.
57
PCP and its variants
Source: T. Bouwmans, Foreground Detection using Principal Component Pursuit: A Survey, under preparation.
How to improve PCP?
Algorithms for solving PCP (17 Algorithms) Incremental PCP (5 papers) Real-Time PCP (2 papers)
Validation for background modeling and foreground detection (3 papers) [ICIP 2012][ICIAR 2012][ISVC 2012]
58
PCP and its variants
Source: T. Bouwmans, Foreground Detection using Principal Component Pursuit: A Survey, under preparation.
59
Validation Background Modeling and Foreground Detection: Qualitative Evaluation
RSL
PCP-EALM
PCP-IADM
PCP-LADM
BPCP-IALM
Original image
Ground truth
PCA
PCP-LSADM
Source: ICIP 2012, ICIAR 2012, ISVC 2012
60
Validation Background Modeling and Foreground Detection : Quantitative Evaluation
F-Measure
Source: ICIP 2012, ICIAR 2012, ISVC 2012
Block PCP gives the best performance!
61
PCP and its application to Background Modeling and Foreground Detection
Recent improvements: BPCP (Tang et Nehorai (2012)) : Spatial but not
incremental and not real time! Recursive Robust PCP (Qiu and Vaswani (2012) ):
Incremental but not real time! Real Time Implementation on GPU (Anderson et al.
(2012) ): Real time but not incremental!
What we can do? Research on real time incremental robust PCP!
62
Conclusion Fuzzy Background Subtraction Background Subtraction via a Discriminative
Subspace Learning: IMMC Foreground Detection via Robust Principal
Component Analysis (RPCA)
Fuzzy Learning Rate Other Discriminative Subspace Learning methods
such as LDA Incremental and real time RPCA
Perspectives
Publications Chapter
T. Bouwmans, “Background Subtraction For Visual Surveillance: A Fuzzy Approach”, Handbook on Soft Computing for Video Surveillance, Taylor and Francis Group, Chapter 5, March 2012.
International Conferences :
F. El Baf, T. Bouwmans, B. Vachon, “Fuzzy Statistical Modeling of Dynamic Backgrounds for Moving Object Detection in Infrared Videos”, CVPR 2009 Workshop, pages 1-6, Miami, USA, 22 June 2009.
F. El Baf, T. Bouwmans, B. Vachon, “Type-2 Fuzzy Mixture of Gaussians Model: Application to Background Modeling”, ISVC 2008, pages 772-781, Las Vegas, USA, December 2008
F. El Baf, T. Bouwmans, B. Vachon, “A Fuzzy Approach for Background Subtraction”, ICIP 2008, San Diego, California, U.S.A, October 2008.
F. El Baf, T. Bouwmans, B. Vachon. " Fuzzy Integral for Moving Object Detection ", IEEE-FUZZY 2008, Hong Kong, China, June 2008.
F. El Baf, T. Bouwmans, B. Vachon, “Fuzzy Foreground Detection for Infrared Videos”, CVPR 2008 Workshop, pages 1-6, Anchorage, Alaska, USA, 27 June 2008.
F. El Baf, T. Bouwmans, B. Vachon, “Foreground Detection using the Choquet Integral”, International Workshop on Image Analysis for Multimedia Interactive Services, WIAMIS 2008, pages 187-190, Klagenfurt, Austria, May 2008.
Fuzzy Background Subtraction
Publications
Journal
D. Farcas, C. Marghes, T. Bouwmans, “Background Subtraction via Incremental Maximum Margin Criterion: A discriminative approach” , Machine Vision and Applications, March 2012.
International Conferences :
C. Marghes, T. Bouwmans, "Background Modeling via Incremental Maximum Margin Criterion", International Workshop on Subspace Methods, ACCV 2010 Workshop Subspace 2010, Queenstown, New Zealand, November 2010.
D. Farcas, T. Bouwmans, "Background Modeling via a Supervised Subspace Learning", International Conference on Image, Video Processing and Computer Vision, IVPCV 2010, pages 1-7, Orlando, USA , July 2010.
Background Subtraction via IMMC
Publications Chapter
C. Guyon, T. Bouwmans, E. Zahzah, “Robust Principal Component Analysis for Background Subtraction: Systematic Evaluation and Comparative Analysis”, INTECH, Principal Component Analysis, Book 1, Chapter 12, page 223-238, March 2012.
International Conferences :
C. Guyon, T. Bouwmans. E. Zahzah, “Foreground Detection via Robust Low Rank Matrix Factorization including Spatial Constraint with Iterative Reweighted Regression”, International Conference on Pattern Recognition, ICPR 2012, Tsukuba, Japan, November 2012.
C. Guyon, T. Bouwmans. E. Zahzah, “Moving Object Detection via Robust Low Rank Matrix Decomposition with IRLS scheme”, International Symposium on Visual Computing, ISVC 2012,pages 665–674, Rethymnon, Crete, Greece, July 2012.
C. Guyon, T. Bouwmans, E. Zahzah, “Moving Object Detection by Robust PCA solved via a Linearized Symmetric Alternating Direction Method”, International Symposium on Visual Computing, ISVC 2012, pages 427-436, Rethymnon, Crete, Greece, July 2012.
C. Guyon, T. Bouwmans, E. Zahzah, "Foreground Detection by Robust PCA solved via a Linearized Alternating Direction Method", International Conference on Image Analysis and Recognition, ICIAR 2012, pages 115-122, Aveiro, Portugal, June 2012.
C. Guyon, T. Bouwmans, E. Zahzah, "Foreground detection based on low-rank and block-sparse matrix decomposition", IEEE International Conference on Image Processing, ICIP 2012, Orlando, Florida, September 2012.
Foreground Detection via RPCA