Activity Monitoring

October 19-20, 1999

DARPADARPADARPADARPA

Bob Bolles, Brian Burns,Marty Fischler, Ravi Gopalan,

Marsha Jo Hannah, Dave ScottSRI International

Rama Chellappa, Yiannis Aloimonos, Doug Ayers,

Ross Cutler, Larry Davis,Azriel Rosenfeld, Chandra Shekhar

University of Maryland

2October 19-20, 1999

Application Challenge

Develop techniques for dramatically increasing the productivity of an aerial video analyst.

3October 19-20, 1999

High-Level Approach

Sensor Multiplexing to Monitor Several Sites “Simultaneously” and Semi-automatically

4October 19-20, 1999

Technical Goalfor Activity Monitoring

Develop techniques to monitor sites, such as cantonment areas and tree lines, from an airborne platform and identify tactically significant activities involving people and vehicles.

Sample Activities:• people entering a forbidden area• people congregating near an embassy• vehicles convoying along a road• people readying a missile for launch

5October 19-20, 1999

Technical Challengesfor Activity Monitoring

• Representation of activities

• Recognition of activities from a moving platform

• Moving object classification

Activity

A large tactical vehicle exiting a hide site (along a tree line). People are often visible guiding the vehicle out.

Starting search

Looking for people

Detect person(s)

Looking for large vehicle

All people leave the FOV

Exit of large vehicle detected

Detect small vehicle

Activity Template

Zoom to a NFOV &aim close to tree line

Move to new pointalong tree line

Detect large vehicle

6October 19-20, 1999

Approach

Task specification•Retrieve or sketch a site model (roads, buildings,…)•Specify the task (what, where, when, & reports/alarms)

Automatic monitoring•Scan the appropriate area •Stabilize the video (MTS -- Sarnoff)•Register the video to the site model (PVR -- Harris)•Detect and track moving objects•Characterize & classify the tracked objects•Recognize activities•Report tactically significant events

AMIS -- Activity Monitoring Integrated Systesm

7October 19-20, 1999

Site Model

Site model

Task specification

Scan area

Stabilize video

Register video

Track objects

Characterize objects

Recognize activities

Report events

Powers Road

Mosby Road

Motorpool

Berm

“Residence” Area

8October 19-20, 1999

Task Specification

Site model

Task specification

Scan area

Stabilize video

Register video

Track objects

Characterize objects

Recognize activities

Report events

Drivers jog to their vehicles

Vehicles drive away

Drivers jog to their vehicles

Motorpool

Residence Area

9October 19-20, 1999

Scan Area

Site model

Task specification

Scan area

Stabilize video

Register video

Track objects

Characterize objects

Recognize activities

Report events

Motorpool

Residence Area

Sensor Field of View

10October 19-20, 1999

Stabilize Video

Site model

Task specification

Scan area

Stabilize video

Register video

Track objects

Characterize objects

Recognize activities

Report events

Raw Video

11October 19-20, 1999

Stabilize Video

Site model

Task specification

Scan area

Stabilize video

Register video

Track objects

Characterize objects

Recognize activities

Report events

Stabilized Video

12October 19-20, 1999

Register Video

Site model

Task specification

Scan area

Stabilize video

Register video

Track objects

Characterize objects

Recognize activities

Report events

Desired field of view

Actual field of view

13October 19-20, 1999

Track Objects

Site model

Task specification

Scan area

Stabilize video

Register video

Track objects

Characterize objects

Recognize activities

Report events

14October 19-20, 1999

Characterize Objects

Site model

Task specification

Scan area

Stabilize video

Register video

Track objects

Characterize objects

Recognize activities

Report events

Object Properties

•Size, velocity, …

•Articulation -- periodicity

(for animate/inanimate)

•Could it be parallax?

•Color, shape, …

•Location in the site

15October 19-20, 1999

Report Events

Site model

Task specification

Scan area

Stabilize video

Register video

Track objects

Characterize objects

Recognize activities

Report events

People moving down Powers Road

Vehicles leavingmotorpool area

People approachingmotorpool area

People enteredmotorpool areaAlert: Battle Group Pullout

16October 19-20, 1999

Primary Contributions

• Representation and recognition of activities (in the context of a site model)

– augmented finite state machines

– dynamic belief networks

• Moving object classification components– parallax analysis

– animate/inanimate classification

– velocity, size, ...

17October 19-20, 1999

Introduction toLive Flight Experiments

18October 19-20, 1999

Activity Monitoring

1. Battle group pullout

2. Battle group return

3. People exiting woods near berm

4. People crossing the road

Berm

“Residence” Area

Activities

Motorpool

19October 19-20, 1999

Activity Templates

Event Primitives– Approaching/Leaving– Gaining-Ground/

Losing-Ground– Entering/Exiting– Moving-inside-region– Temporal durations

Combinations– Boolean operations– Sequences– Graphs

Starting search

Looking for people

Detect person(s)

Looking for large vehicle

All people leave the FOV

Exit of large vehicle detected

Detect small vehicle

Activity Template

Zoom to a NFOV &aim close to tree line

Move to new pointalong tree line

Detect large vehicle

20October 19-20, 1999

Site Model Sketching

21October 19-20, 1999

Video Registration

Image

World

22October 19-20, 1999

Activity Analysisin World Coordinates

Image

World

23October 19-20, 1999

Moving Object Detection

Raw video fields

Raw differences

AND’d differences

Image N

24October 19-20, 1999

Parallax Versus Independent Motion

25October 19-20, 1999

Animate/Inanimate

Periodicity analysis

26October 19-20, 1999

Align and scale objects

Compute similarity matrix S

Template fit peaks of S

Track objects

Autocorrelate S

T r ead m il l

1 0 20 3 0 40 5 0 60 7 0 80 9 0 1 00

1 0

2 0

3 0

4 0

5 0

6 0

7 0

8 0

9 0

1 00

Tre admill

10 20 30 40 5 0 60

10

20

30

40

50

60

Periodicity Analysisfor Classifying Objects as

Animate or Inanimate

27October 19-20, 1999

Parallax Detection

Flagged as being locally consistent with “motion parallax”

28October 19-20, 1999

AM’s Windows

29October 19-20, 1999

Stabilization Params

Metadata

MTS-Ground

Multiple Target Surveillance

Precision VideoRegistration

Raw Video (analog)

CAGS-Ground

CAGS-Air

Ground Station

Activity Monitoring

Air-Ground Partitionfor 1999

30October 19-20, 1999

Battle Group Pullout

1. Battle group pullout

2. Battle group return

3. People exiting woods near berm

4. People crossing the road

Activities

Drivers jog to their vehiclesDrivers jog to their vehicles

Vehicles drive away

31October 19-20, 1999

Battle Group Return

Vehicles return & park

Drivers walk back to residence

1. Battle group pullout

2. Battle group return

3. People exiting woods near berm

4. People crossing the road

Activities

32October 19-20, 1999

People Exiting Woods near Berm

People Exit Tree Line

1. Battle group pullout

2. Battle group return

3. People exiting woods near berm

4. People crossing the road

Activities

33October 19-20, 1999

People Crossing Road

People Exit Tree Line and cross the road

1. Battle group pullout

2. Battle group return

3. People exiting woods near berm

4. People crossing the road

Activities

34October 19-20, 1999

PreliminaryEvent Statistics

E vent Trials Successes Success Rate

Approaching 8 8 100%

Entering 14 13 93%

Moving Inside 16 16 100%

Exiting 13 10 77%

Leaving 8 8 100%

Totals 59 55 93%

• Results from 2 flights with high contrast imagery

35October 19-20, 1999

PreliminaryWhole Vignette Statistics

Vignette Trials Successes Success Rate

Battle Group Pullout 5 5 100%

Battle Group Return 5 4 80%

People Exit Woods 1 1 100%

People Cross Road 6 3 50%

People Stealing Vehicles 3 3 100%

Totals 20 16 80%

• Results from 2 flights with high contrast imagery

36October 19-20, 1999

Summary

Accomplishments:• AMIS – Activity Monitoring Integrated System• Activity Templates – an initial representation for activities• An initial technique for recognizing activities based on augmented

finite state machines• An extension to dynamic belief networks to activity recognition• A technique for identifying moving objects due to motion parallax • A technique for classifying moving objects as animate or inanimate• A semi-automatic video registration technique• A realtime moving object detection technique

Increase the productivity of an image analyst by a factor of 10 to 15 by multiplexing a high-performance sensor and automatically identifying potentially significant activities.

Goal:

37October 19-20, 1999

Evaluation of‘99 Accomplishments

Moving object classification -- Components only

Sensor Control -- manual versus computer-controlled

HCI -- primarily on PC, not integrated into CAGS-Ground

38October 19-20, 1999

Plans for ‘00

• Represent & recognize more complex activities, such as checkpoint monitoring

• Call PVR for video registration

• Place sensor under computer-control (based on MTS results)

• Integrate moving object classification

• Integrate the HCI into CAGS-Ground