Distributed Video Data Fusion, Analysis, and Mining for Video Surveillance Applications* Edward...

transcript

Distributed Video Data Fusion, Distributed Video Data Fusion, Analysis, and Mining for Video Analysis, and Mining for Video Surveillance Applications* Surveillance Applications*

Edward Chang2 and Yuan-Fang Wang1

Department of Electrical and Computer Engineering2

Department of Computer Science1

University of CaliforniaSanta Barbara, CA 93106

*Supported in part by NSF Career, ITR, IDM, and Infrastructure grants, and a gift from Proximex Corp.

Problem Statement Video surveillance with

Multiple cameras Mobile, wireless networks Online data processing Intelligent, computer-assisted content analysis

Focus of current work Event Sensing for

detection representation, and Recognition of motion events

Sensor Network Management for Bandwidth and power resource conservation

Potential Applications and Needs

Applications Emergency search and rescue in natural disaster Deterrence of cross-border illegal activities Reconnaissance and intelligence gathering in

digital battlefields Needs

Rapid deployment, dynamic configuration, and continuous operations

Robust and real-time data fusion and analysis Intelligent event modeling and recognition

TtZtYtXt ))(),(),(()( P

Ttytxt ))(),(()( 111 p

Internet

Slave station

Masterstation

Validation Scenario

Research and Development Framework

Event detection Far-field coordination and update Near-field sensor data fusion

Event representation Hierarchical – multiple levels of detail Invariant – insensitive to incidental changes

Event recognition Temporally correlated event signature Imbalanced training set

Event Detection: Near-field Sensor Data Fusion

Sensing coordination and intelligent data fusion

Two-level hierarchy of Kalman filter

Bottom level (feed forward) Summarize trajectories in local

state vectors Merge state vectors from multiple

cameras through registration parameters

Top level (feed backward) Fill in missing or occluded

trajectory pieces Camera pose & frame rate control

)()0( tz)()( tiz )()1( tmz

XTxworldreal

)0()0(

)0()0( xTX

realworld

XTxworldreal

)1()1(

realworld m xTXInternet

Master fusion station

Slave stationSlave station

Slave station

Event Detection: Far-field Coordination and Update

Minimizing Bandwidth and power consumption under pre-specified accuracy constraints

Dual Kalman filters Update necessary only when

predications diverge Cache dynamic algorithms instead of

static data

Event Representation

Hierarchical Multiple levels of description

Syntactic level Semantic level

Invariant Descriptors unaffected by incidental changes of

environmental factors and camera pose Consequences

Be able to perform both “intra-class” and “inter-class” recognition

Recognize syntactic similarity (the same trajectory) and semantic similarity (the same type of trajectory)

Event Representation: Syntactic Level

Normalization against View point (Affine or

perspective) Speed

To derive an invariant signature

Event Representation: Semantic Level Segmentation based on acceleration Segment characterization Markov chain representation

?0P ierP no

?0V oyes constant? r

Stoppedyes no

Constantvelocity

Right spiral

yes no

constant?

?|| oVP

Left half turn

yes no

Slow down

yes no

Right half turn

0)( zoVP

yes no

Right outwardturn

0)( zoVP

yes no

Rightinwardturn

0 oVP 0 oVP

Left outwardturn

Leftinwardturn

yes no yes no

0/ dtd

yes no yes no

Right turn

Left turn

yes no

0/ dtd

Left spiral

yes no

Quickaccelerate

yes no

Quickstart

constant?

?|| oVP

Left half Turn w.acc

yes no

yes noEmergency stop

yes no

Right half turn w.acc

0)( zoVP

yes no

0)( zoVP

yes no

Rightoutwardturn w acc

0 oVP 0 oVP

yesno yes

0/ dtd

yes no yes no

Left turn w. acc

yes no

0/ dtd

yes no

0/|| dtd r

Right half turn w.decel

0/|| dtd r

Left half Turn w.decel

yesno no

0/|| dtd r

yesno no

0/|| dtd r

yesno no

Rightoutwardturn w decel

Rightinwardturn w acc

Rightinwardturn w decel

Leftoutwardturn w acc

Leftoutwardturn w decel

Leftinwardturn w acc

Leftinwardturn w decel

0/|| dtd r

yesno no

0/|| dtd r

yesno no

Left turn w. decel

Rightturn w. acc

Rightturn w. decel

Left turn w. acc

Left turn w. decel

RightTurn w. acc

Rightturn w. decel

Event Representation: Semantic Level (cont.)

Left half turn

Left half turn w. acc

Left half turn w. decel

Left outwardspiral

Left outward spiral w. acc

Left outward spiral

w. decel

Left inwardspiral

Left inward spiral w. acc

Left inward spiral

w. decelConstant velocity

Speed up

Slow down

Left half turn

Left outwardspiral

Left outward spiral

w. decel

Left inwardspiral

Left inward spiral

w. decel Constant velocity

Speed up

Slow down

Left half turn

Left outwardspiral

Left outward spiral

w. decel

Left inwardspiral

Left inward spiral

Speed up

Slow down

Left half turn

Left outwardspiral

Left outward spiral

w. decel

Left inwardspiral

Left inward spiral

Speed up

Slow down

Event Recognition: Sequence Data Learning

Similarity measurement difficult Sequence data with temporal correlation may

not have a vector space representation However, kernel methods (e.g., SVM) are

applicable No vector space representation OK But with feature space representation

Use DP algorithm for feature space distance metric Use hierarchical kernel recognition and fusion

Event Recognition: Imbalanced Data Set Negative samples significantly

outnumber positive samples Bayesian risk associated with

false negative significantly outweighs false positive

Adaptive conformal mapping at decision boundary

Event Recognition: Statistical Modeling

HMM is expensive to build

Not all behaviors are structured (e.g., loitering behaviors)

It may not be necessarily to understand individual activities before recognizing interaction

Distinguish interaction patterns Following Following-and-

gaining Stalking

Experimental Results: Syntactic Matching

Experimental Results: Semantic Indexing

Experimental Results: Biased Learning

=TP/(TP+FN)

=TN/(TN+FP)

threshold

penalty

Experimental Results: Statistical Learning

Results

Relevant Publications

Many details are omitted Sensor registration (spatial and temporal) Object tracking (Kalman and multi-state) Power management and routing

1. L. Jiao, G. Wu, Y. Wu, E. Y. Chang, and Y. F. Wang, “The Anatomy of A Multi-Camera Video Surveillance System,'' to appear in the ACM Multimedia System Journal.

2. K. Wu, J. Long, D. Han, and Y. F. Wang, “Human Activity Detection and Recognition for Video Surveillance,” Proceedings of IEEE International Conference on Multimedia Computing and Systems, 2004.

3. Edward Chang and Yuan-Fang Wang, "Toward Building a Robust and Intelligent video Surveillance System: A Case Study," (invited paper) Proceedings of the IEEE Multimedia and Expo Conference, Taipei, Taiwan, 2004.

4. R. Rangaswami, Z. Dimitrijevic, K. Kakligian, Edward Chang, and Yuan-Fang Wang, "The SfinX Video Surveillance System," Proceedings of the IEEE Multimedia and Expo Conference, Taipei, Taiwan, 2004.

5. G. Wu, Y. Wu, L. Jiao, Y. F. Wang, and E. Y. Chang, `”Multi-camera Spatio-temporal Fusion and Biased Sequence-data Learning for Security Surveillance,'' Proceedings of ACM Multimedia Conference, Berkeley, CA, 2003.

6. K. Wu, J. Long, D. Han, and Y. F. Wang, “Real-Time Multi-person Tracking in Video Surveillance,” Proceedings of the Pacific Rim Multimedia Conference, Singapore, 2003.

7. Y. Wu, L, Jiao, G. Wu, E. Chang, and Y. F. Wang, “Invariant Feature Extraction and Biased Statistical Inference for Video Surveillance,” Proceedings of the IEEE International Conference on Advanced Video and Signal-based Surveillance, Miami, FL, 2003.

Focus of This Seminar

Video-based face tracking, modeling and recognition

Human activity and interaction analysis

Video-Based Face Tracking & Recognition

Image-based Image normalization Feature selection Face recognition

Video-based Face region detection Tracking Face modeling and recognition

Difficulties

Quality of video is low Large illumination, pose variation, occlusion

Face images are small Compared to still image-based system

Model construction and fitting Generic vs. personal-specific 2D vs. 3D

Proposed Approach: Resolution Enhancement

Exploit multiple image frames and spatial coherency Single camera super-resolution (digital zoom) Multi-camera (master-slave) face region detection and

zooming (optical zoom) Need feature appearance (PCA + LDA) and

geometrical relations

General Framework: Visual Servoing

A Feedback control mechanism Reference and real signals are computed

from images

- J-1 Camera Control +

External Disturbance

New Image

FeatureDection

Referencesignal

Realsignal

Errorsignal

Controlsignal

Master-Slave Combo Setup

slaveslaveslaveworldworldmastermaster

slaveworldslaveworldmastermasterworldworldslaveslave

worldworldmastermaster

pTTpPTp

),,(),,(

fslavep

),,,( slaveslaveworld zfT

worldmasterT

Mater: Anatomy-Guided Face Modeling

Face region localization based on anatomy Face region detection based on skin color

segmentation Face region modeling based on ellipse fitting Face region tracking using mean-shift tracker

worldmasterT

Slave: Master-Guided Zooming

fslavep

),,,( slaveslaveworld zfT

worldmasterT

What’s Next?

View-based recognition Frontal-view detection Multi-frame evidence aggregation 3D model (?)

Single Camera Super resolution

Multiple, spatially-coherent frames as down-sampled, low-resolution (LR) images of original high-resolution (HR) images

Mathematically

IIIIII

nITBDI

Three components: Spatial registration function

(T) Blurring function (B) Down-sampling function

(D) c: down-sampling factor

Spatial Registration Function

Modeled as affine transform Capture translation, rotation, and zooming In reality, only translation motion has been

successfully demonstrated

xxxk cba

Blurring Function

Modeled as Gaussian kernel Caveats:

point spread function (blurring) function may not be known and is wave-length dependent

Diffraction effect induces ripples and is better modeled with Besel functions

Numerical Solution

Large system of equations Require preconditioning

Not sure that it will work in the real world Simpler mechanism (e.g., bi-linear

interpolation) exists with inferior performance

Optical zoom instead of digital zoom

Schedule 9/29: overview 10/6: Dan: face recognition overview 10/13: no meeting (research travel) 10/20: Dr. Kang 10/27: 11/3: 11/10: 11/17: 11/24:

Video-based face modeling and recognition Super resolution

Multiple images Space-time

Human activity/interaction analysis

Video-based face modeling and recognition Super resolution

Multiple images Space-time

Human activity/interaction analysis

Distributed Video Data Fusion, Analysis, and Mining for Video Surveillance Applications* Edward...

Documents