Candidate: Francesco Castaldo Advisor: Francesco Palmieri

Outline

• Goals and tools

• First year

– Target tracking using factor graphs

• Second year

– Situation awareness with Bayesian models

• Third year

– Cross-view semantic matching

– Visual prediction and knowledge transfer

Goals

• Surveillance and analysis of complex scenarios – Ports – Parking lots – Busy streets – Etc.

• Applications – Target tracking (I year) – Situation awareness (II year) – Visual prediction (III year)

• Visual input data – The knowledge we can

extract from images or videos

Tools

• Probabilistic graphical models

• Hidden Markov Models

• Factor graphs

• Dynamic Bayesian networks

• Machine learning • Nearest neighbor

• RANSAC

First year

• Where:

– Second University of Naples (SUN)

• Application:

– Target Tracking using Factor Graphs and Multi-camera systems

Motivation

• Target tracking – Reconstruct the state of

moving targets • Position • Velocity

• Issues – Occlusions – Noisy sensors – Inaccurate models – Data association

• “Which is my measurements?”

6

Sensors

7

• Cameras – Model

• Pinhole model (2D-3D)

• Homography (2D-2D)

– Calibration • Need for point

correspondences

• Other sensors – flexible framework

FFG framework

9

• Normal factor graph (FFG):

• Edges are the variables

• Nodes are the factors.

• Given directions we can define forward (f) and backward (b) messages.

• Each factor block describes the conditional probability function that maps the input into the output variable.

• The fusion bus merge bi-directionally all the information coming from the system.

The model

Results

Results

• Bi-directional message propagation results in a more accurate estimation

– Smoothing on the trajectory

• Noisy calibration stage

– the errors affect the range

of the estimates

Conclusions

• What we have accomplished – Innovative target tracking framework that is

• Modular

• Flexible

• Robust

– Investigated what is the impact of bad camera calibration on tracking

• What to do next – Multiple target tracking

• By propagating mixtures of Gaussians

Second year

• Where: – University of Genoa

• Collaboration: – ISIP40 Lab led by prof.

Regazzoni

• Project: – Bayesian Analysis of

Behaviors and Interactions for Situation Awareness

Motivation

• Many accidents happened in recent years.

• Why?

Motivation

• Monitoring by human operators – Great number of sensors as

cameras, radars, etc.

• What is missing? – Situation awareness

• i.e. the ability to analyze the behavior of the agents in the scene in order to detect abnormal events in time.

• We build a model to detect unusual situations – Support human operators

Assumptions

• Fixed scenario

• Single target – Behavior analysis (raw

analysis)

• Varying number of (possibly)-interacting target (context information) – Pairwise interactions

analysis (refinement)

The framework

1. Trajectory data are used to extract a topological map of the area.

2. Probabilistic models for behaviors and interactions are first trained with normal data and using the map of step 1.

3. The models are then used in real time for situation awareness with new data.

Map creation

• Topological map of the area under surveillance – Partitioning of the area in

zones of different size and shape

• Why?

– Reduce the computational complexity of the algorithm

– Detect composite and complex situations

Partitioning of the map

• Regular grid

– Not very smart

• Intelligent map

– We have the trajectories, therefore we want a map that is:

• Coarse in areas less explored

• Fine in areas much explored

Probabilistic models

• Behavior analysis

– What the target is doing

• Interaction analysis

– One-on-one interactions with other targets

Behavior Analysis

Behavior Analysis

Interaction Analysis

Results

• The creation of a monitoring system which accomplishes two main tasks:

– Acknowledge normal traffic condition • for each target the system has to be capable of identifying the type

of action carried out by the agent at that time.

– Detect uncommon/dangerous situations in time • In this way the human operators have the time to schedule potential

countermeasures (alarm, evacuation, etc.).

Results

Port of Amsterdam Simulator of trajectories

ITM map Voronoi map

Results

• Behavior analysis – Three situations:

• Horizontal with moderate speed (red) [normal]

• Vertical with moderate speed (blue) [normal]

• Horizontal with high speed (magenta) [dangerous]

• Interaction analysis – Two situations:

• Normal interaction (green) [normal]

• Abnormal interaction (malfunctioning of the ship moving in vertical direction) [dangerous]

Conclusions

• What we have accomplished

– Bayesian framework for situation awareness in complex scenarios

– The models are computationally cheap and flexible

• If new situations arise, a new model can be added to the framework

• Operator in the loop

Third year

• Where: – Stanford University

• Collaboration: – CVGL Lab led by prof.

Savarese

• Projects: – Cross-view semantic

matching

– Visual prediction and knowledge transfer

Motivation

• A moving pedestrian appears in the following scene

• We ask ourselves: what he is going to do in the near future?

Introduction Learn how to predict the most likely

activities

Visual prediction

• Two goals

– Path prediction

• Given trajectory data of a scene, design a model for path forecasting on the same scene

– Knowledge transfer

• Given trajectory data and semantics of N training scenes, transfer our knowledge to a new unseen scene for path forecasting

Navigation map

• We overlay an uniform grid on the scene

• For each patch we learn a minimal set of information for path forecasting

– Popularity score

– Routing score

– Histogram of Direction (HoD)

– Histograms of Speed (HoS)

(a) Input scene

(b) “Navigation maps”

HoD and HoS

Navigation map

Popularity scores Routing scores “Velocity” vectors

• We learn these maps from trajectory data of the scene

The model

• Bayesian model

• No observations

– Purely in prediction

– We need only the initial condition and the navigation maps

Path prediction

Semantic transfer

• We transfer the navigation maps to new similar scenes – Scenes sharing the same semantic layout will probably

have similar navigation maps

Semantic transfer

• We use a semantic descriptor to capture the semantic layout of each patch

Semantic Transfer

• For each patch in the new scene we transfer our knowledge of the K most similar patches from any training scene

• We build the navigation maps for a new unseen scene

Transfer

Results

Results

Conclusions

• Bayesian path forecasting

• Almost first attemp of modeling robust knowledge transfer

• What to do next

– This was only human-scene interaction

• Add human-human interaction to have a general framework

Publications

Publications

Publications

T H A N K S!