Post on 05-Oct-2020
transcript
Candidate: Francesco Castaldo Advisor: Francesco Palmieri
Outline
• Goals and tools
• First year
– Target tracking using factor graphs
• Second year
– Situation awareness with Bayesian models
• Third year
– Cross-view semantic matching
– Visual prediction and knowledge transfer
Goals
• Surveillance and analysis of complex scenarios – Ports – Parking lots – Busy streets – Etc.
• Applications – Target tracking (I year) – Situation awareness (II year) – Visual prediction (III year)
• Visual input data – The knowledge we can
extract from images or videos
Tools
• Probabilistic graphical models
• Hidden Markov Models
• Factor graphs
• Dynamic Bayesian networks
• Machine learning • Nearest neighbor
• RANSAC
First year
• Where:
– Second University of Naples (SUN)
• Application:
– Target Tracking using Factor Graphs and Multi-camera systems
Motivation
• Target tracking – Reconstruct the state of
moving targets • Position • Velocity
• Issues – Occlusions – Noisy sensors – Inaccurate models – Data association
• “Which is my measurements?”
6
Sensors
7
• Cameras – Model
• Pinhole model (2D-3D)
• Homography (2D-2D)
– Calibration • Need for point
correspondences
• Other sensors – flexible framework
FFG framework
9
• Normal factor graph (FFG):
• Edges are the variables
• Nodes are the factors.
• Given directions we can define forward (f) and backward (b) messages.
• Each factor block describes the conditional probability function that maps the input into the output variable.
• The fusion bus merge bi-directionally all the information coming from the system.
The model
Results
Results
• Bi-directional message propagation results in a more accurate estimation
– Smoothing on the trajectory
• Noisy calibration stage
– the errors affect the range
of the estimates
Conclusions
• What we have accomplished – Innovative target tracking framework that is
• Modular
• Flexible
• Robust
– Investigated what is the impact of bad camera calibration on tracking
• What to do next – Multiple target tracking
• By propagating mixtures of Gaussians
Second year
• Where: – University of Genoa
• Collaboration: – ISIP40 Lab led by prof.
Regazzoni
• Project: – Bayesian Analysis of
Behaviors and Interactions for Situation Awareness
Motivation
• Many accidents happened in recent years.
• Why?
Motivation
• Monitoring by human operators – Great number of sensors as
cameras, radars, etc.
• What is missing? – Situation awareness
• i.e. the ability to analyze the behavior of the agents in the scene in order to detect abnormal events in time.
• We build a model to detect unusual situations – Support human operators
Assumptions
• Fixed scenario
• Single target – Behavior analysis (raw
analysis)
• Varying number of (possibly)-interacting target (context information) – Pairwise interactions
analysis (refinement)
The framework
1. Trajectory data are used to extract a topological map of the area.
2. Probabilistic models for behaviors and interactions are first trained with normal data and using the map of step 1.
3. The models are then used in real time for situation awareness with new data.
Map creation
• Topological map of the area under surveillance – Partitioning of the area in
zones of different size and shape
• Why?
– Reduce the computational complexity of the algorithm
– Detect composite and complex situations
Partitioning of the map
• Regular grid
– Not very smart
• Intelligent map
– We have the trajectories, therefore we want a map that is:
• Coarse in areas less explored
• Fine in areas much explored
Probabilistic models
• Behavior analysis
– What the target is doing
• Interaction analysis
– One-on-one interactions with other targets
Behavior Analysis
Behavior Analysis
Interaction Analysis
Results
• The creation of a monitoring system which accomplishes two main tasks:
– Acknowledge normal traffic condition • for each target the system has to be capable of identifying the type
of action carried out by the agent at that time.
– Detect uncommon/dangerous situations in time • In this way the human operators have the time to schedule potential
countermeasures (alarm, evacuation, etc.).
Results
Port of Amsterdam Simulator of trajectories
ITM map Voronoi map
Results
• Behavior analysis – Three situations:
• Horizontal with moderate speed (red) [normal]
• Vertical with moderate speed (blue) [normal]
• Horizontal with high speed (magenta) [dangerous]
• Interaction analysis – Two situations:
• Normal interaction (green) [normal]
• Abnormal interaction (malfunctioning of the ship moving in vertical direction) [dangerous]
Conclusions
• What we have accomplished
– Bayesian framework for situation awareness in complex scenarios
– The models are computationally cheap and flexible
• If new situations arise, a new model can be added to the framework
• Operator in the loop
Third year
• Where: – Stanford University
• Collaboration: – CVGL Lab led by prof.
Savarese
• Projects: – Cross-view semantic
matching
– Visual prediction and knowledge transfer
Motivation
• A moving pedestrian appears in the following scene
• We ask ourselves: what he is going to do in the near future?
Introduction Learn how to predict the most likely
activities
Visual prediction
• Two goals
– Path prediction
• Given trajectory data of a scene, design a model for path forecasting on the same scene
– Knowledge transfer
• Given trajectory data and semantics of N training scenes, transfer our knowledge to a new unseen scene for path forecasting
Navigation map
• We overlay an uniform grid on the scene
• For each patch we learn a minimal set of information for path forecasting
– Popularity score
– Routing score
– Histogram of Direction (HoD)
– Histograms of Speed (HoS)
(a) Input scene
(b) “Navigation maps”
HoD and HoS
Navigation map
Popularity scores Routing scores “Velocity” vectors
• We learn these maps from trajectory data of the scene
The model
• Bayesian model
• No observations
– Purely in prediction
– We need only the initial condition and the navigation maps
Path prediction
Semantic transfer
• We transfer the navigation maps to new similar scenes – Scenes sharing the same semantic layout will probably
have similar navigation maps
Semantic transfer
• We use a semantic descriptor to capture the semantic layout of each patch
Semantic Transfer
• For each patch in the new scene we transfer our knowledge of the K most similar patches from any training scene
• We build the navigation maps for a new unseen scene
Transfer
Results
Results
Conclusions
• Bayesian path forecasting
• Almost first attemp of modeling robust knowledge transfer
• What to do next
– This was only human-scene interaction
• Add human-human interaction to have a general framework
Publications
Publications
Publications
T H A N K S!