Video Analysis

Content Extraction

1

Dennis Moellman, VACE Program Manager

VACE Executive Brief

for MLMI

Video Analysis

Content Extraction

2

Briefing Outline

• Introduction • Phase II• Evaluation• Technology Transfer• Phase III• Conclusion

Video Analysis

Content Extraction

3

Introduction

• ARDA – Advanced Research and Development Activity– A high-risk/high-payoff R&D effort sponsored by US

DoD/IC

• ARDA taking a new identity– In FY2007 under the DNI

• Report to: ADNI(S&T)

– Renamed: Disruptive Technology Office

• VACE – Video Analysis and Content Extraction– Three Phase initiative begun in 2000 and ending 2009

• Winding down Phase II• Entering into Phase III

What is ARDA/DTO/VACE?

Video Analysis

Content Extraction

4

Context

• Problem Creation:– Video is an ever expanding source of imagery and open source

intelligence such that it commands a place in the all-source analysis.

• Research Problem:– Lack of robust software automation tools to assist human

analysts:• Human operators are required to manually monitor video signals• Human intervention is required to annotate video for indexing

purposes• Content based routing based on automated processing is lacking• Flexible ad hoc search and browsing tools do not exist

• Video Extent:– Broadcast News; Surveillance; UAV; Meetings; and

Ground Reconnaissance

Video Exploitation Barriers

Video Analysis

Content Extraction

5

Research Approach

• Research Objectives:– Basic technology breakthroughs– Video analysis system components– Video analysis systems– Formal evaluations: procedures, metrics and data sets

• Evaluate Success:– Quantitative Testing

Metric Current NeedAccuracy <Human >>HumanSpeed >Real time <<Real time

– Technology Transition• Over 70 technologies identified as deliverables• 50% have been delivered to the government• Over 20 undergoing government evaluation

Video Exploitation

Video Analysis

Content Extraction

6

Management Approach

Management Philosophy – NABC

• N – Need

• A – Approach

• B – Benefit• C – Competition

Geared for Success

Video Analysis

Content Extraction

7

Language/User Technology

Visualization

EnhancementFilters

Source Video

RecognitionEngine

UnderstandingEngine

ExtractionEngine

Reference

01101010101011111010010101101101010101

0110100110

01101011Intelligent

Content Services

Concept Applications

Interests

System View

Video Analysis

Content Extraction

8

Motion Analysis

Evaluation

Inte

llige

nt

Cont

ent

Serv

ices

Summarization

Video Browsing

Content-based RoutingAdvanced query/retrieval using Q&A technologies

Video Monitoring

Video Mining

Indexing

Change Detection

Con

tent

Extr

action Object Recognition

Mensuration

Scene Modeling

Event Recognition

Object Modeling

Simple Event Detection

Event Understanding

Complex Event Detection

Object/Scene ClassificationObject Detection & Tracking

Enab

ling

Tech

nol

ogie

s

Image Enhancement/Stabilization

Multi-modal fusion

Event Ontology

Event Expression Language

Camera Parameter Estimation

Automated Annotation Language

Integrity Analysis

Phase 2 Phase 3Phase 1 Future

Motion Analysis

Filtering

VACE Interests

Technology Roadmap

Video Analysis

Content Extraction

9

Funding

FY06 Allocations FY07 Allocations4%

12%

20%

64%36%

39%

4%

10%

11%

Commitment to Success

Video Analysis

Content Extraction

10

Phase II

• Researcher Involvement: – Fourteen contracts– Researchers represent a cross section of industry

and academia throughout the U.S. partnering to reach a common goal

• Government Involvement: – Tap technical experts, analysts and COTRs from

DoD/IC agencies– Each agency is represented on the VACE

Advisory Committee, an advisory group to the ARDA/DTO Program Manager

Programmatics

Video Analysis

Content Extraction

11

Phase II

Demographics

Virage

MIT

ColumbiaUniv.

Univ. of Maryland (2)

AFIT

Univ. of Illinois-Urbana-Champaign (2)

PurdueUniv.

Univ. ofChicago

Univ. ofWashington

SRI

Univ. of Southern

California / Info.Science Inst.

Univ. of Central Florida

GeorgiaInst. Of Tech.

TelcordiaTechnologies

Sub Contractors (14)

Univ. of SouthernCalifornia

IBM T. J.Watson Center

BoeingPhantom

Works

CarnegieMellon

Univ. (2) (Robotics Inst.)

(Informedia) .

Wright StateUniv.

TASC

Sarnoff Corp (2)

BBN

Salient Stills

Alphatech

Univ. of Illinois-Urbana-Champaign

Prime Contractors (14)

Univ. of Maryland

Video Analysis

Content Extraction

12

Phase II

ProjectsTitle Organization Principal Investigator

Foreign Broadcast News Exploitation

ENVIE: Extensible News Video Information

Exploitation

Carnegie Mellon University

Howard Wactlar

Reconstructing and Mining of Semantic Threads Across Multiple Video Broadcast

News Sources using Multi-Level Concept Modeling

IBM T.J. Watson Research Center / Columbia Univ.

John Smith /

Prof. Shih-Fu Chang

Formal and Informal Meetings

From Video to Information: Cross-Modal Analysis of

Planning Meetings

Wright State University

VaTech/AFIT / Univ. of Chicago / Purdue

Univ. / Univ. of Illinois-Urbana-Champaign

Francis Quek / Ronald Tuttle /

David McNeill & Bennett Bertenthal / Thomas Huang / Mary Harper

Event Recognition from Video of Formal and Informal

Meetings Using Behavioral Models and Multi-modal

Events

BAE Systems/ MIT / Univ. of Maryland /

Virage

Victor Tom/ William Freeman & John Fisher /

Yaser Yacoob & Larry Davis / Andy Merlino

Video Analysis

Content Extraction

13

Phase II

Projects

Title Organization Principal Investigator

Abstraction and Inference about Surveillance Activities

Video Event Awareness Sarnoff Corporation /

Telcordia Technologies Rafael Alonso /

Dimitrios Georgakopoulos

Integrated Research on Visual Surveillance

University of Maryland Larry Davis,

Yiannis Aloimonos & Rama Chellappa

UAV Motion Imagery

Adaptive Video Processing for Enhanced Object and Event Recognition in UAV Imagery

Boeing Phantom Works (Descoped to UAV data

collection)Robert Higgins

Task and Event Driven Compression (TEDC) for UAV

Video Sarnoff Corporation Hui Cheng

Ground Reconnaissance Video

Content and Event Extraction from Ground Reconnaissance

Video

TASC, Inc. / Univ. of Central Florida / Univ. of

California-Irvine

Sadiye Guler / Mubarak Shah / Ramesh Jain

Video Analysis

Content Extraction

14

Phase II

Projects

Title Organization Principal Investigator

Cross Cutting / Enabling Technologies

Probabilistic Graphical Model Based Tools For

Video Analysis

University of Illinois, Urbana - Champaign

Thomas Huang

Automatic Video Resolution Enhancement

Salient Stills, Inc. John Jeffrey Hunter

Robust Coarse-to-Fine Object Recognition in

Video

CMU/Pittsburgh Pattern Recognition

Henry Schneiderman & Tsuhan Chen

Multi-Lingual Video OCR BBN Technologies /

SRI International John Makhoul / Greg

Myers

Model-based Object and Video Event Recognition

USC - Institute for Robotics and

Intelligent Systems / USC - Information Sciences Institute

Ram Nevatia, Gerard Medioni & Isaac Cohen /

Jerry Hobbs

Video Analysis

Content Extraction

15

Evaluation

• Programmatic:– Inform ARDA/DTO management of

progress/challenges

• Developmental:– Speed progress via iterative self testing– Enable research and evaluation via essential data and

tools – build lasting resources

• Key is selecting the right tasks and metrics– Gear evaluation tasks to research suite– Collect data to support all research

Goals

Video Analysis

Content Extraction

16

Evaluation

The Team

NIST

USF

Video Mining

Video Analysis

Content Extraction

17

Evaluation

NIST Process

Formal Evaluation

Determine Sponsor Requirements

Assess required/existing

resources

Develop detailed plans with

researcher input

Dry-Run shakedown

Technical Workshopsand reports

Recommendations

PlanningPlanning ProductsProducts ResultsResultsEvaluation Plan

Task Definitions

Protocols/Metrics

Rollout Schedule

Data Identification

Evaluation Resources

Training Data

Development Data

Evaluation Data

Ground Truth and other metadata

Scoring and Truthing Tools

Video Analysis

Content Extraction

18

Evaluation

NIST Mechanics

Video Data

Algorithms

Evaluation

Results

Annotation

System Output

Ground Truth

Measures

Video Analysis

Content Extraction

19

Evaluation

2005-2006 Evaluations

Evaluation TypeEvaluation TypeDetectionDetection TrackingTracking RecognitionRecognition

DomainDomain PP FF VV TT PP FF VV TT PP FF VV TT

Meeting RoomMeeting Room xx xx       xx xx                  

Broadcast NewsBroadcast News xx xx xx xx xx xx xx xx          xx

UAVUAV xx    xx    xx    xx               

SurveillanceSurveillance xx xx xx    xx xx xx               

Ground ReconGround Recon                                    

P = Person; F = Face; V = Vehicle; T = Text

Video Analysis

Content Extraction

20

Evaluation

• Evaluation Metrics:– Detection: SFDA (Sequence Frame Detection

Accuracy) • Metric for determining the accuracy of a detection

algorithm with respect to space, time, and the number of objects

– Tracking: STDA (Sequence Tracking Detection Accuracy)

• Metric for determining detection accuracy along with the ability of a system to assign and track the ID of an object across frames

– Text Recognition: WER (Word Error Rate) and CER (Character Error Rate)

• In-scene and overlay text in video

• Focused Diagnostic Metrics (11)

Quantitative Metrics

Video Analysis

Content Extraction

21

Evaluation

Phase II Best Results

Spatial Detection and Tracking Error

0%

10%

20%

30%

40%

50%

60%

70%

80%

Text(Broacast

News)

Face(Broacast

News)

Face(MeetingRoom)

Hand(MeetingRoom)

Person(MeetingRoom)

MovingVehicle(UAV)

Evaluation Tasks

Per

cen

t E

rro

r

Detection

Tracking

Video Analysis

Content Extraction

22

Evaluation

PPATT UIUC UCF0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(20

%T

) S

FD

A

Boxplot of (20%T) SFDA scores: Face in BNews

Sites

Face Detection: BNews (Score Distribution)

Video Analysis

Content Extraction

23

Evaluation

COLIB SRI_1 SRI_2 CMU_L0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(20

%T

) S

FD

A

Sites

Boxplot of (20%T) SFDA Scores: EngText in BNews

Text Detection: BNews (SFDA Score distribution)

Video Analysis

Content Extraction

24

Evaluation

• Benefit of open evaluations– Knowledge about others’ capabilities and community feedback– increased competition -> progress

• Benefit of evaluation workshops– Encourage peer review and information exchange, minimize

“wheel reinvention”, focus research on common problems, venue for publication

• Current VACE-related open evaluations– VACE: Core Evaluations – CLEAR: Classification of Events, Activities, and Relationships – RT: Rich Transcription– TRECVID: Text Retrieval Conference Video Track– ETISEO: Evaluation du Traitment et de l’Interpretation de

Sequences Video

Open Evaluations and Workshops -- International

Video Analysis

Content Extraction

25

Evaluation

Expanded Source Data

Task Sub-conditionConference

MeetingsSeminar Meetings Surveillance

Broadcast News

Studio Poses UAV

3D Sing Per Track Video   CHIL        

  Audio   CHIL        

  Audio+Video   CHIL        

2D Multi Per Detect   VACE         VACE

2D Multi Per Track   VACE CHIL VACE     VACE

Person ID Video   CHIL        

  Audio   CHIL        

  Audio + Video   CHIL        

2D Face Det   VACE     VACE    

2D Face Track   VACE CHIL   VACE    

Head Pose Est     CHIL     CHIL  

Hand Detect   VACE          

Hand Track   VACE     VACE    

Text Detect         VACE    

Text Track         VACE    

Text Recognition         VACE    

Vehicle Detect       VACE     VACE

Vehicle Track       VACE     VACE

Feature Extract         TRECVID    

Shot Boundary Detect         TRECVID    

Search         TRECVID    

Acoustic Event Detection     CHIL        

Environment Class     CHIL        

Video Analysis

Content Extraction

26

Evaluation

2005 2006 2007 2008A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D

Evaluation Schedule VACE Phase 2 VACE Phase 3TRECVID CLEAR RT

NIST/DTO VACE

VACE Phase 2 Core

Evaluation Series 1 (Text, Face in Meeting, BNews)

Evaluation Series 2 (Hand in Meeting, Vehicle in UAV)

Evaluation Series 3 (Person in Meeting)

Evaluation Series 4 (Person/Vehicle in Surveillance)

VACE Phase 3 Core

Evaluation Series 5, 6, 7, 8

TRECVID

(Search and Extraction)

CLEAR

(Person/Face/Pose in Meeing)

(Extraction/Localiazation)

Rich Transcription

(English/Arabic Text in BNews)

(Multi-Modal Recognition)

ETISEO

4/30 6/30

5/96/6 11/8

Report Results

8/249/23 11/8

Report Results

9/810/3 11/8

Report Results

6/28

Report Results

2/23/2 4/5

Report Results

8/2 9/610/16

Report Results

2/43/114/22

Report Results

7/148/2310/2

Report Results

4/12 9/3011/14

Report Results

4/7 9/20 11/13

Report Results

4/6 9/20 11/13

Report Results

4/7 9/19 11/11

Report Results

2/1 3/94/8

Report Results

3/14/25/11

Report Results

3/34/4 5/9

Report Results

5/4

Report Results

4/306/1 7/5

Report Results

4/306/67/9

Report Results

1/2

Cycle 1

5/1 5/8

Cycle 2

11/17 12/8

Seminar

Schedule

Video Analysis

Content Extraction

27

TECH TRANSFER

Purpose: Move technology from lab to operation• Technology Readiness Activity

– An independent repository for test and assessment– Migrate technology out of lab environment– Assess technology maturity – Provide recommendations to DTO and researchers

DTO Test and Assessment Activities

Video Analysis

Content Extraction

28

Level Definitions Entry Condition Contractor Activity

1 Basic principles observed and reported Some peer review of ideas Reporting on basic idea

2  Technology concept and/or application formulated

Target applications are proposed Speculative work; invention

3 Analytical and experimental critical function and/or characteristic proof of concept

Algorithms run in contractor labs and basic testing is possible (internal, some external may be possible)

Doing Analytical studies with weakly integrated components

4  Component/breadboard validation in lab Proof of concept exists; test plans exist; external testing is possible

Low fidelity integration of components

5 Component/breadboard validation in relevant environment

Integrated system functions outside contractor lab; some TRA tests completed

Working with realistic situations

6 System/subsystem model or prototype in demonstration in relevant environment

IC/DoD users identified; target environment defined; simulated testing possible

Demonstrating engineering (software qualities) feasibility

7 System prototype demo in operational environment

Test lab trials in simulated environment completed; installed in operational environment

Completing the product

8 Actual system completed and qualified through test and demonstration

Product completed; Test lab trial completed successfully

Releasing the product; Repairing minor bugs

9 Actual system proven through successful mission operations

Proven value-added in an operational environment

Repairing minor bugs; noting proven operational results

DoD Technology Readiness Levels (TRL)

TECH TRANSFER

Video Analysis

Content Extraction

29

DTOControl

DTOInfluence

DOD Technology Risk Scale

HIGH LOWRISK

Technology Transfer

Use in assessing project’s• Technology maturity• Risk level• Commercialization potential

6 7

8 9

4 5

1 2 3

IC/DODTest

Facility(s)

Info-XTest

FacilityContractorTest

Facility

UNCLASSIFIED

UNCLASSIFIEDCLASSIFIED

UNCLASSIFIEDCLASSIFIED

Production

Applying TRL

Video Analysis

Content Extraction

30

Technology Transfer

TRL Assesment

0

1

2

3

45

6

7

8

9

Ba

ckg

rou

nd

Su

btr

act

ion

Sta

bili

zatio

n

Ob

ject

Tra

ckin

g_

Sta

tCa

me

ra

Imp

ort

an

ceM

ap

Ge

ne

rato

r

Mu

ltila

yerM

osa

icG

en

era

tor

Eve

ntE

stim

atio

n

Ma

rve

l

Info

rme

dia

ST

AR

T

Gu

time

PIQ

UA

NT

-II

MP

SG

Op

inio

nF

ind

er

Kin

evi

s

Ge

oT

ime

TR

L

Assesment prior to delivery

Current assesment

Projected at end of Contract

TRA Maturity Assessments

Video Analysis

Content Extraction

31

Phase III BAA

• Contracting Agency: DOI, Ft. Huachuca, AZ– DOI provides COR– ARDA/DTO retain DoD/IC agency COTR’s and add more

• Currently in Proposal Review Process– Span 3 FY’s and 4 CY’s– Remains open thru 6/30/08

• Funding objective: $30M over program life– Anticipate to grow in FY07 and beyond

• Address the same data source domains as Phase II• Will conduct formal evaluations• Will conduct maturity evaluations and tech transfer

Programmatics

Video Analysis

Content Extraction

32

Phase III BAA

• Emphasis on technology and system approach– Move up technology path where applicable– Stress ubiquity

• Divided into two tiers:– Tier 1: One year base with option year

• Technology focus• Open to all – US and international• More awards for lesser funding

– Tier 2: Two year base with option year(s)• Comprehensive component/system level initiative• Must be US prime• Fewer awards for greater funding

Programmatics

Video Analysis

Content Extraction

33

Phase III BAA

Schedule2005 2006

Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct

Planning Schedule Milestone Major Task

Activity

Scope Planning

Planning Group Meetings

BAA RFP

First Draft

Second Draft

Third Draft

Final Draft

Contracting Final Review

BAA

Announcement

Questions/Proposals Due

Proposal Evaluation

Source Selection Meeting

Funding Recommendations

Contract Awards

Phase 3 Kickoff Workshop

6/14 6/30 8/31

8/2 9/27 10/18 11/7

DOI

11/22

7/18 2/2

7/18 8/2

Six weeks after kickoff

9/1 9/30

Six weeks after first draft

10/10 11/4

Completed prior to Fall Workshop

11/21 12/2

Sent to Contracting Office

12/2

Draft Final BAA

12/16 1/8 1/27

Final BAA

12/15 6/30

12/15

Draft BAA Posting

1/20

Brief

2/1

Final BAA Posting

1/6

Questions/Comments

3/3

Proposals

3/6 4/144/25

4/26-27

4/28

5/1 6/30

8/9-10

Video Analysis

Content Extraction

34

Summary

• VACE is interested in:– Solving real problems with risky, radical

approaches– Processing multiple data domains and multimodal

data domains– Developing technology point solutions as well as

component/system solutions– Evaluating technology process– Transferring technology into user’s space

Take-Aways

Video Analysis

Content Extraction

35

Conclusion

• Invitations:– Welcome to participate in VACE Phase III– Welcome to participate in VACE Phase III Evaluations

Potential DTO Collaboration

Video Analysis

Content Extraction

36

Contacts

Dennis Moellman, Program Manager

Phones: 202-231-4453 (Dennis Moellman) 443-479-4365 (Paul Matthews) 301-688-7092

(DTO Office)800-276-3747 (DTO

Office)

FAX: 202-231-4242 (Dennis Moellman)301-688-7410 (DTO

Office)

E-Mail: dennis.moellman@dia.mil (Internet Mail) pmmatth@nsa.gov

Location: Room 12A69 NBP #1Suite 6644

9800 Savage RoadFort Meade, MD 20755-6644