1
Detecting and Distinguishing Nuances of Anomaly in Machine Perception Systems
Josef Kittler Centre for Vision, Speech and Signal Processing
Dept. Electronic Engineering, University of Surrey Guildford, UK
Coauthors: F Yan, T de Campos, D Windridge, W Christmas, M Osman, A Khan, N Faraji Davar and J Illingworth
Support by EPSRC is gratefully acknowledged
2
Outline
Background Introduction to anomaly detection Prior art Critique of conventional anomaly
detection Proposed anomaly detection system Application to tennis video annotation
number of players anomaly out-of-play anomaly
Conclusions
Vision system design
Architecture Vision system modules Training data Learning system parameters System integration System optimisation
3
Advanced concepts in vision systems design
High level scene interpretation Ability to adapt and acquire new
competences Anomaly detection Acquisition of domain knowledge Context classification
4
Tennis video interpretation
5
6
Application to sports video annotation
Aim is to interpret tennis video from the observed visual events
The states are: 1st serve 2nd serve Ace Rally Point award
7
Tennis annotation system 15 modules
videosource
on/off switch
data flow
frame
lens correctionde−interlace &
homography
corner tracking/ homographies
constructmosaic
mosaic
shotclassification
shotClass
closeup play
shotdetection
shotBoundary
for ball tracking
ballCandidate
finds candidates
detect eventsin 3D world
threeDCues
reasoningbased on rulesof tennis
highLevel
HMM eventdetector
HMMEDetection
anomalyDetection
detects anomalies inball events
anomalyAnalysis
compares anomalyanalysis with ground truth
tennisanotation
white linefinding
whiteLines
calibrationcamera
projection
ok
proj
serveDetection
detects serve pointfilters action results
separates foregroundfor ball tracking
ballForeground
corner tracking /
homographiesalternate
skipHomogs
ballInterpolation
Interpolate ball trackwhen off top of screen
ballTracking
track ball anddetect ballevents in 2D
creates table ofanomaly vs. noise
anomalyVsNoise
foreground
separatesforeground forplayer tracking
track players
playerTrack
classifieshit/serve/non−hitactions
playerActions
playerCount
in no. of playersdetects anomalies
Low−level Ball trackingMosaicing /projection Player tracking High−level
8
Tennis annotation system
A system that “understands” tennis In the sense: video in -> score out High level reasoning relies on low level
processing Ball tracking is crucial
shot analysis
court detection
ball tracking
player tracking
events detection HMM
camera score
low level high level
Contextual interpretation
Objects Object
configurations
9
o i
o j
ω α ω β
Scene Graph Model Graph
Sensory data
World model
Scene model
Examples in tennis video Object dynamics Global view of static environment Spatial relationship of objects Visual event evolution
(grammars, Hidden Markov models )
Static background removal
11
Ball/player event detection
12
13
Event Detection I
Visual event detection
14
• Simplified for ease of training • Error correcting Viterbi algorithm • Point awarding model only • Future extensions to game and set models
Match evolution model
15
The tennis annotation system
Testing on tennis doubles
How will the system respond to tennis doubles?
What mechanisms are needed for the system to extend its competence to the task of interpreting tennis doubles?
16
ACASVA Project
Existing machine perception systems largely no ability to adapt no ability to extend competence/transfer
knowledge ACASVA aims to develop mechanisms for
cognitive bootstrapping, i.e. ability to adapt detectors transfer knowledge adapt interpretation processes acquire new competences
17
18
Background
Mechanisms needed for system
competence extension Anomaly detection Visual event – anomaly association Model adaptation Context classifier
Introduction to anomaly
Anomaly – an important notion in human
understanding of the environment deviation from normal order or rule
Many synonyms signifying different nuances rarity, irregularity, incongruence,
abnormality, unexpected event, novelty, innovation, outlier
19
Classical anomaly model
In science/engineering prove disprove hypothesis fault detection outdated model requires adaptation
Conventional mathematical model outlier of a distribution empirical distribution deviates from
the model distribution
20
x
Prior art in anomaly detection
Edgeworth (1888) Hundreds of papers Many approaches
statistical, NN, classification, clustering, information theoretic, spectral
Excellent surveys Markou&Singh (SP 2003, statistical, neural) Hodge&Austin (AI Review 2004) Agyemang&Barker&Alhajj (Int Data Anal 2006) Chandola&Banerjee&Kumar (ACM Surveys 2010) Saligrama&Konrad&Vodoin (SPM2010, video)
21
Classical model
22
23
Tennis annotation system 15 modules
videosource
on/off switch
data flow
frame
lens correctionde−interlace &
homography
corner tracking/ homographies
constructmosaic
mosaic
shotclassification
shotClass
closeup play
shotdetection
shotBoundary
for ball tracking
ballCandidate
finds candidates
detect eventsin 3D world
threeDCues
reasoningbased on rulesof tennis
highLevel
HMM eventdetector
HMMEDetection
anomalyDetection
detects anomalies inball events
anomalyAnalysis
compares anomalyanalysis with ground truth
tennisanotation
white linefinding
whiteLines
calibrationcamera
projection
ok
proj
serveDetection
detects serve pointfilters action results
separates foregroundfor ball tracking
ballForeground
corner tracking /
homographiesalternate
skipHomogs
ballInterpolation
Interpolate ball trackwhen off top of screen
ballTracking
track ball anddetect ballevents in 2D
creates table ofanomaly vs. noise
anomalyVsNoise
foreground
separatesforeground forplayer tracking
track players
playerTrack
classifieshit/serve/non−hitactions
playerActions
playerCount
in no. of playersdetects anomalies
Low−level Ball trackingMosaicing /projection Player tracking High−level
Anomaly detection & machine perception
Multiple models Discriminative classifiers Ambiguity of interpretation Contextual reasoning Hierarchical representation Data quality Model pruning
24
Classical notion of anomaly
25
Distribution outlier
Different aspects of anomaly
26
Different aspects of anomaly
27
-Distribution drift -Novelty detection
Data quality
28
Data quality
29
Incongruence/unexpected event
30
Magritte’s La duree poignard Model base pruning
Computational efficiency Hierarchical representation
o i
o j
ω α ω β
Scene Graph Model Graph
Conventional anomaly detection
31
Proposed anomaly detection system
32
Incongruence detection
Detecting differences between observations and expectations (anomaly, rare event, incongruence)
Basic principle – comparison of outputs of weak and strong classifiers (Ketabdar et al 2007)
Dirac Project (Burget et al 2008, Weinshall et all [2009-2012])
Exemplified by out-of-vocabulary word detection Phoneme recognizer (weak classifier) HMM speech recognizer (strong, contextual classifier)
33
Proposed anomaly detection system
34
Nuances of anomaly
No anomaly Noisy measurement Unknown object Corrupted
measurement Congruent labelling Unknown structure Spurious
measurement errors
35
Unexpected structural component
Unexpected structural component & structure
Measurement model drift
Next challenge
Tennis to badminton
36
#players detection
0 1 2 3 4 5 6 7 80
0.2
0.4
0.6
0.8Singles
A03WSA03MSJ09WS
0 1 2 3 4 5 6 7 80
0.2
0.4
0.6
0.8
moving agents
Doubles
A08WDU06WD
41
# player anomaly detection
0 2 4 6 8 100
10
20
30
40
50
60
70
80
90
100
buffer size (in shots)
dete
cted
ano
mal
ies
(% o
f sho
ts)
Train with A03MS+J09WS
A08WD (doubles)U06WD (doubles)A03WS (singles)
42 0 2 4 6 8 10
0
10
20
30
40
50
60
70
80
90
100
buffer size (in shots)
dete
cted
ano
mal
ies
(% o
f sho
ts)
Train with A03WS+A03MS
A08WD (doubles)U06WD (doubles)J09WS (singles)
0 2 4 6 8 100
10
20
30
40
50
60
70
80
90
100
buffer size (in shots)
dete
cted
ano
mal
ies
(% o
f sho
ts)
Train with A03WS+J09WS
A08WD (doubles)U06WD (doubles)A03MS (singles)
Out-of-play anomaly detection
43
Out-of-play anomaly detection
44
Decision confidence filter
Anomalous ball event detection
45
Contextual postprocessing
Bayesian surprise
46
Measure of incongruence
contextual aposteriori probability
non-contextual aposteriori probability
o i
o j
Bayesian surprise has undesirable properties
Modified measure of surprise
Modified measure of surprise in 2 class
case
47
Out-of-play anomaly
2008 Ground Truth Anomaly Triggering Events (27/176)
0
1
2
3
4
5
6
7
8
9
10
11
48
Bootstrapping system
49
Context detection
Contextual annotation
Anomaly detection
Low level event
detection
Knowledge base
Rule base learning
Conclusions
Novel anomaly detection system Incongruence detector Decision confidence filter Data quality assessment Computationally efficient outlier detector
Applied to anomaly detection in tennis video analysis
Demonstrated merits of all the mechanisms deployed by the system
50
