Date post: | 11-Feb-2017 |
Category: |
Engineering |
Upload: | francesco-flammini |
View: | 131 times |
Download: | 0 times |
PERFORMANCE EVALUATION OF VIDEO ANALYTICS
FOR SURVEILLANCE ON-BOARD TRAINS
Context and Objectives
ACIVS’13, October 30th 2013, Poznan, Poland
Low level Evaluation
Real-time video surveillance is used in public transportation, including on-board metro trains, to support human operators in control rooms.
Video Content Analytics (VCA) is effective when the performance is such to reduce
false alarms under appropriate acceptability thresholds. Reducing the causes of false alarms by fine tuning the VCA is particularly important in
on-board applications, that are more prone to disturbances with respect to fixed installations due to more critical operating conditions (e.g. constrained camera field of view, frequent light changes, vibrations, occlusions, etc.).
Accurate performance evaluation is essential to decide about:
Which alarms to activate depending on scenarios How to set algorithm parameters in each scenario Comparison between commercial and open-source implementations is also important to evaluate the added value provided by the former in terms of achievable performance.
Black Box Evaluation
Valentina Casola, Mariana Esposito, Francesco Flammini, Nicola Mazzocca, Concetta Pragliola
Contributions
Evaluation of Commercial Off-The-Shelf (COTS) VCA system
Low Level Black Box
Alarm Performance Evaluation
Comparison with Open source
System
Frame and Object Based Metrics
Ground Truth values compared with Algorithm Result
Computed by specific tool developed in Matlab
Object Based Metrics
Consider the whole trajectory of each
object in the scene and preserves its
lifetime
Correct Detected Track (CDT)
Track Detection Failure (TDF)
False Alarm Track (FAT)
Track Fragmentation (TF)
ID Change (IDC)Temporal Overlap
Spatial Overlap
Performance Evaluation ToolAlgorithm Result
Ground Thruth
Tot Tos
Metrics
(15%)
Frame Based Metrics
Measure the performance on individual
frames of a video stream and do not
consider the preservatio. The blobs are
evaluated in their size and location and
compared with GT.
True Positive (TP)
False Negative (FN):
Fragmentation
ecall
False Positive (FP)
Merging
Precision
0
50
100
150
200
250
300
350
400
Oc
cu
rre
nc
ies
#AR TP FN FP TF IDC
Object-Based Results
COTS
Open-Source
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
Pe
rfo
rma
nc
e In
de
x
Accuracy FAR PP FNR
Object-Based Performance
COTS
Open-Source
False Alarm Rate FPTP
FP
FAR
Detection rateFNTP
TP
DR
Positive PredictionFPTP
TP
PP
False Negative RateTPFN
FN
FNR
Fragmentation IndexFRAGM
TPFI
Merge IndexMERGE
FPOBJ AR FM
♦ 1510 objects in 600 frames, 8 GT tracks ♦ COTS system detects less objects but it is far more reliable regarding false positives (84 vs 2168) ♦ FA of 6% instead of 41% (the latter
clearly unacceptable). ♦ COTS object Fragmentation is less then one half w.r.t. Open source.
Metrics COTS Open
TP 1307 1324
FN 203 186
FP 84 2168
Metrics COTS Open
TP 7 8
FN 1 0
FP 9 213
TF 11 17
IDC 0 0
♦ 1510 objects in 600 frames, 8 GT tracks ♦ COTS FP and PP reveal higher reliability ♦ Same consideration for TF (important for event persistence).
Performance IndicesTest of COTS system on real
vehicle for light railwails and
tramways
False Alarm IndexRealAlarms#
sFalseAlarm#FAI
Probability Of DetectionPDTPFN
TP
POD
over it detects camera occlusionblurring with paint or obstacles.amper it detects the manumission of the camera by moving it.top it detects still objects in the scene.Presence it detects objectspeople moving in the scenerowd it raises an alarm when the scene is overcrowded.
ALARMS