Post on 21-Mar-2018
transcript
National Science Foundation Engineering Research Center
National Science Foundation Engineering Research Center
iWatch: iWatch: Intelligent Surveillance at Scale byConnecting-the-Dots in Space and Time!
Farnoush Banaei-KashaniViterbi School of Engineering
University of Southern CaliforniaLos Angeles, CA 90089
banaeika@usc.edu
National Science Foundation Engineering Research Center
Intelligent Surveillance
Surveillance [sɜːˈveɪləns]: Obtaining information and knowledge about events of interest through observation, investigation, and analysis for the sake of understanding
Intelligent Surveillance : IT + Surveillance
Pictures courtesy of rotoconcept.com
National Science Foundation Engineering Research Center
Intelligent Surveillance at Scale!
Home Scale
Campus Scale
National Scale
Human-in-the-Loop Surveillance
National Science Foundation Engineering Research Center
• Human-Deeply-in-the-Loop Surveillance!– Efficiency: Limited human processing power limits capabilities of
the system in handling, e.g.: • Numerous heterogeneous input feeds• High input and event rate• Large number of events of interest
– Accuracy: Human is error-prone!
– Cost: Waste of resources
Limitation of the Existing Systems
National Science Foundation Engineering Research Center
Our Vision: Geo-Immersive Surveillance
• Support numerous input feeds of various modalities
• Let machine capture all “incidents”
• And piece together incidents in space and time to generate “event” scenarios
• Forensic analysis in historic data
• Real-time monitoring of current input feeds
• Event prediction of potential expected (and unexpected!) events
National Science Foundation Engineering Research Center
Applications
• Large-scale Urban Securitye.g., campus incidents
• Military Intelligencee.g., counter-insurgency (COIN) war
• Plant Disaster Managemente.g., oil spills
• National Securitye.g., terrorist attacks
National Science Foundation Engineering Research Center
iWatch: A Prototype System
• Application: Campus Security Tool for USC’s Department of Public Safety (DPS)
• Platform: iCampus.USC• Multidisciplinary Team:
– App: Carol Hayes, David Beeler– Event Detection: Ram Nevatia (tracking),
Gerard Medioni (face detection)– Spatiotemporal Query and Analysis: Cyrus
Shahabi, Farnoush Banaei-Kashani– Visualization: Ulrich Neumann
National Science Foundation Engineering Research Center
Current Status
• Input: Video feed from 25 PTZ cameras• Mode: Forensic Analysis
National Science Foundation Engineering Research Center
Research Challenges
• Advanced video event detection– Allow for sparse coverage– Detect from mobile video (unknown parameters)– Devise event ontology
• Spatiotemporal Query and Analysis– Spatiotemporal modeling of the events– Spatiotemporal indexing of the events– Support for uncertain data
• Visualization– Rich and realistic modeling using multiple complementary data
sources (e.g., aerial as well as street LiDAR)
Talk by Ram Nevatia
National Science Foundation Engineering Research Center
Future Research Plan /Collaboration Opportunities
• Closing the loop by personalized information provision
• Mobile video: communication and spatiotemporal modeling
• Advanced video event detection:– Face recognition– Human and social behavior analysis
• Extending the supported data modalities beyond video (text, sensor readings, audio, etc.)
• Have other applications?• Have complementary ideas and technology?
Talk by Daniel Goldberg
Talk by Roger Zimmermann
National Science Foundation Engineering Research Center
Sustainability
• Potential Opportunities– USC– Government Research Grants
• Security and Intelligence: DHS, NGA, ARDA, NIJ
• Defense: NURI, MURI
– Industry• Cash gift• Research Proposals• Integrating industry products into the iWatch prototype• License iWatch to other universities & small cities• Spin-off company!
: Google: HP
National Science Foundation Engineering Research Center
Q & A
National Science Foundation Engineering Research Center
Telling What-is-What in Video
Ram Nevatia, Professor
University of Southern California
National Science Foundation Engineering Research Center
Objectives and Approach
• Objectives– Detect and track objects of interest in a video– Infer activities of interest– Detect anomalies
• Approach– Assume calibrated, stationary video camera– Detect objects based on shape and motion– Tracking based on detections– Activity inference based on structured models
National Science Foundation Engineering Research Center
• Part-based representation; allows for detection in presence of occlusions– Requires joint reasoning of inter-occluding humans
• Part detectors use local features, e.g. edgelets• Detectors are cascades of classifiers learned by
AdaBoost procedure• Tracking consists of linking detection responses
– Hierarchical associations– Learning of affinity functions
Shape Based Human Detection and Tracking
National Science Foundation Engineering Research Center
Human Detection Examples
National Science Foundation Engineering Research Center
Tracking Example (iWatch)
National Science Foundation Engineering Research Center
Tracking Result Video (TREVid)
National Science Foundation Engineering Research Center
Two Approaches of Activity Detection
• Bottom-up Approach– Track pose/position, infer events from tracks– More general and easier to extend but bottom-up analysis
is not robust• Top-down Approach
– Apply event models directly to low-level features extracted from the videos
– Poses come as a by-product of event recognition• Simultaneously produces pose tracks and action recognition
(STAR)
– More robust but makes a “closed world” assumption
National Science Foundation Engineering Research Center
Event Detection ResultsDrop and Pick up Event
Left Luggage Event (Video)
National Science Foundation Engineering Research Center
Activity Detection Result Video
National Science Foundation Engineering Research Center
Future Direction
• Develop methods for detecting and tracking objects that humans interact with, e.g. a briefcase, left luggage etc.
• Scene objects can provide valuable context but detection of such objects is difficult– Campus models may provide some of this
• Define a class of important activities for iWatch• Integrate the object and event detection capabilities
into the iWatch system
National Science Foundation Engineering Research Center
Q & A
National Science Foundation Engineering Research Center
Daniel Goldberg, Kaveh Shahabi, John WilsonSpatial Sciences Institute
College of Letters, Arts & SciencesUniversity of Southern California
Los Angeles, CA 90089-0374dwgoldbe@usc.edu
http://spatial.usc.edu
Targeted Trojan Alerts: Targeted Trojan Alerts: Enabling Dynamic LocationEnabling Dynamic Location--Based Evacuation StrategiesBased Evacuation Strategies
National Science Foundation Engineering Research Center
Targeted Alerts Motivation
TROJANSALERT
“When an emergency occurs, authorized senders can instantly notify you using TROJANS ALERT.
TROJANSALERT is your connection to real-time updates, instructions on where to go, what to do, or what not to do, who to contact and other important information.”
National Science Foundation Engineering Research Center
Emergency Notification
KAP
GER
DNI
OHE
RTH
RRI
DRBSuspicious package, PSA,3rd floor NE corner.Stay away from area
National Science Foundation Engineering Research Center
Broadcast Messages
Suspicious package, PSA,3rd floor NE corner.Stay away from area
Glendale
Park La BreaSanta Monica
Culver City
LAXDowney
LBI New Jersey
National Science Foundation Engineering Research Center
Geo-Targeting Messages
KAP
GER
DNI
OHE
RTH
RRI
DRB
Personalize messages•Location of the individual•Relation to the threat
KAP•Exit building •Head north and west toward Vermont and Jefferson
OHE•Exit building •Head east toward the center of campus
CAL•Remain inside
ISI•Avoid travelling to UPC
National Science Foundation Engineering Research Center
Targeted Message
You are in the danger zone. Please follow the route to a safe place immediately.
User-Specific Information
Risk
High
Low
National Science Foundation Engineering Research Center
System Operation
End User
Registered Smart Phone
DPS Admin
Trojan Alert Server
ThreatIdentification
Broadcast Message
Report Location
Threat Notification
National Science Foundation Engineering Research Center
Research Questions
• Soliciting Participation– Mobile monitoring and Privacy– Market penetration and Incentives
• Risk and Population Surface Construction– Location-based (GPS – know they are there)– Context-based (Classes – supposed to be there)
• Dynamic Routing– Real-time– Dynamic risk, environment, population– Varying spatial scales
But its for your own good.We swear..
National Science Foundation Engineering Research Center
The Team
• Dan Goldberg– USC Spatial Sciences Institute
• Kaveh Shahabi– PhD Student, USC Computer Science
• John Wilson– USC Spatial Sciences Institute
National Science Foundation Engineering Research Center
QuestionsConcerns
Suggestions
Thanks!
??
????
??
??
??
??
????
National Science Foundation Engineering Research Center
Roger ZimmermannNational University of Singapore
Singapore 117417rogerz@comp.nus.edu.sg
http://geovid.org
GeoVidGeoVidManaging Mobile Video in Managing Mobile Video in
Spatiotemporal SpaceSpatiotemporal Space
Seon Ho KimUniversity of Southern California
Los Angeles, CA 90089seonkim@usc.edu
http://imscwww.usc.edu
National Science Foundation Engineering Research Center
Motivation
• Trends– User-generated video content is growing rapidly.– Mobile devices make it easy to capture video.
• Challenge– Video is still difficult to manage and search.
• Geo-Referenced Video– Location and direction information
can now be collected through anumber of sensors(e.g., GPS and compass).
National Science Foundation Engineering Research Center
Overview of Approach
1. Viewable scene modeling2. Video and meta-data acquisition3. Indexing, querying, and presentation of results
1)
d2) 3)
National Science Foundation Engineering Research Center
Scene Modeling
• Describe the video stream through a sequence of field-of-views (FOV).
2D 3D
FOVScene(P,d,,R) FOVScene(P,d,,,R)
National Science Foundation Engineering Research Center
GeoVid Acquisition
iPhone AppiPhone App Android AppAndroid App
National Science Foundation Engineering Research Center
Spatio-Temporal Search
<-117.010, 46.725><-117.013, 46.725><-117.013, 46.728><-117.010, 46.728>
.
.
.
.
Search for videos that capture the given trajectory
Search for the videos of
“Kibbie Dome”
<-117.010, 46.725><-117.013, 46.725><-117.013, 46.728><-117.010, 46.728>
National Science Foundation Engineering Research Center
Query Execution
Object X
Camera location
• Spatio-temporal queries to find relevant video segments:“Find videos that show object X from t1to t2.”
• Goal:– Reduce
irrelevant video segments (i.e., reduce user browsing time)
National Science Foundation Engineering Research Center
Ex.: Spato-Temporal Range Query
Video 2
Video 1
Queryarea
National Science Foundation Engineering Research Center
Search and Results: 2D http://geovid.org/Query.html
National Science Foundation Engineering Research Center
Search and Results: 3D
National Science Foundation Engineering Research Center
Thank YouThank You
Other Team MembersOther Team Members• Sakire Arslan Ay• Beomjoo Seo• Jia Hao• Guanfeng Wang• Ma He• Shunkai Fang• Lingyan Zhang
http://geovid.orghttp://geovid.orgFurther information at:
rogerz@comp.nus.edu.sgseonkim@usc.edu
National Science Foundation Engineering Research Center
Q & A
National Science Foundation Engineering Research Center
End of the iWatch SessionThanks!