1
PRIMAPerception Recognition and Integration
for Observing and Modeling Activity
James L. Crowley, Prof. I.N.P. GrenobleAugustin Lux, Prof. I.N.P. Grenoble
Patrick Reignier, MdC. Univ. Joseph FourierDominique Vaufreydaz, MdC UPMF
2
The PRIMA Group Leaders
Doms, Jim, Patrick and Augustin
3
The PRIMA Group Members
Trombinoscope
4
The PRIMA Group, May 2006
Permanents :James L. Crowley, Prof. I.N.P. GrenobleAugustin Lux, Prof. I.N.P. GrenoblePatrick Reignier, MdC. U.J.F. Dominique Vaufreydaz, MdC. UPMF.
Assistante : Caroline Ouari (INPG)
Contractual EngineersAlba Ferrer, IE INRIAMathieu Langet, IE INPG
5
The PRIMA Group, May 2006
Doctoral Students : Stan Borkowski (Bourse EGIDE)Chunwiphat, Suphot (Bourse Thailand)Thi-Thanh-Hai Tran (Bourse EGIDE)Matthieu Anne (Bourse CIFRE - France Telecom)Olivier Bertrand (Bourse ENS Cachan)Nicolas Gourier (Bourse INRIA)Julien Letessier (Bourse INRIA)Sonia Zaidenberg (Bourse CNRS - BDI)Oliver Brdiczka (Bourse INRIA)Remi Emonet (Bourse MENSR)
6
Plan for the Review
1) Presentation of Scientific ProjectObjectivesResearch Problems and ResultsBilan 2003 - 2006Evolutions for 2007-2010
7
Objective of Project PRIMA
Develop the scientific and technological foundations for context aware, interactive
environments
Interactive Environment: An environment capable of perceiving, acting,
communicating, and interacting with users.
8
Experimental Platforme : FAMEAugmented Meeting Environment
8 Cameras7 Steerable1 fixed, wide angle
8 Microphones(acoustic Sensors)
6 Biprocessors (3 Ghz)3 Video Interaction Devices (Camera-projector pairs)
January 06: Inauguration of new Smart Environments Lab (J 104)
9
Augmented Meeting Environment
QuickTime™ and a decompressor
are needed to see this picture.
10
Research Problems
• Context-aware interactive environments • New forms of man-machine interaction (using
perception)• Real Time, View Invariant, Computer Vision• Autonomic Architectures for Multi-Modal Perception
11
Research Problems
• Context-aware interactive environments • New forms of man-machine interaction (using
perception)• Real Time, View Invariant, Computer Vision• Autonomic Architectures for Multi-Modal Perception
12
Software Architecture for Observing Activity
Sensors and actuators: Interface to the physical world.Perception and action: Perceives entities, Assigns entities to roles. Situation: Filter events, Describes relevant actors and props for services. (User) Services: Implicit or explicit. Event driven.
User Services
Sensors, Actuators, CommunicationsLogical Sensors, Logical Actuators
Perceptual ComponentsSituation Modeling
Ont ol ogy Ser ve r, Ut il it i es
13
Situation Graph
Situation: An configuration of entities playing roles
Configuration: Set of Relations (Predicates) over entities. Entity: Actors or ObjectsRoles: Abstract descriptions of Persons or objects
A situation graph describes a state space of situationsand the actions of the system for each situation
Situation-1Situation-3
Situation-2Situation-4
Situation-5
Situation-6
Situation Graph
14
Situation and Context
Basic Concepts:
Property: Any value observed by a processEntity: A “correlated” set of propertiesComposite entity: A composition of entitiesRelation: A predicate defined over entitiesActor: An entity that can act.Role: Interpretation assigned to an entity or actor
Situation: A configuration of roles and relations.
15
Situation and Context
Role: Interpretation assigned to an entity or actorRelation: A predicate over entities and actorsSituation: An configuration of roles and relations.
A situation graph describes the state space of situationsand the actions of the system for each situation
Approach: Compile a federation of processes to observe the roles (actors and entities) and relations that define situations.
16
Objective: Automatic acquisition of situation models.
Approach: Start with simple sterotypical model for scenarioDevelop using Supervised Incremental Learning
Recognition: Detect Roles with Linear ClassifiersRecognize Situation using probablisitic model
Acquiring Situation Models
17
Video Acquisition System V2.0
Process SupervisorSituation Modeling
SteerableCamera 1
Wide AngleCamera
Event Bus
VocalActivityDetector
New SlideDetection
Face Detection
SpeakerTracker
Audio-VisualComposition
New Person
Detection
CameraAudienceCameraStreaming Video
MPEG
Mic
Mic
Projector
Camera
18
Audio-Visual Acquisition System
QuickTime™ and a decompressor
are needed to see this picture.
Version 1.0 - January 2004
19
Research Problems
• Context-aware interactive environments • New forms of man-machine interaction (using
perception)• Real Time, View Invariant, Computer Vision• Autonomic Architectures for Multi-Modal Perception
20
Steerable Camera Projector Pair
21
QuickTime™ and a decompressor
are needed to see this picture.
22
Portable Display Surface
QuickTime™ and a decompressor
are needed to see this picture.
23
Rectification by Homography
For each rectified pixel (x,y), project to original pixel and compute interpolated intensity
(x, y) (x', y')
24
Real Time Rectification for the PDS
25
Luminance-based button widget
S. Borkowski, J. Letessier, and J. L. Crowley. Spatial Control of Interactive Surfaces in an Augmented Environment. In Proceedings of the EHCI’04. Springer, 2004.
26
Striplet – the occlusion detector
dxdytyxyx t ),,L(),(f)(R gain
Gain
x
x
y
0),(f gain dxdyyx
27
Striplet – the occlusion detector
x
y
0 R
28
Striplet – the occlusion detector
x
y
0 R
29
Striplet-based SPOD
SPOD – Simple-Pattern Occlusion Detector
30
Projected Calculator
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
31
Research Problems
• Context-aware interactive environments • New forms of man-machine interaction (using
perception)• Real Time, View Invariant Computer Vision• Autonomic Architectures for Multi-Modal Perception
32
Chromatic Gaussian Basis
0
k
GxL
GC1
GC2
GxC1
GxC2
GxxL
GxyL
GyyL
Normalized in Scale and Orientation to Local Neighborhood
33
Real Time, View Invariant Computer Vision
Results• Scale and orientation normalised Receptive Fields computed at
video rate. (BrandDetect system, IST CAVIAR)• Real time indexing and recognition (Thesis F. Pelisson)
• Robust Visual Features for Face Detection (Thesis N. Gourier)
• Direct Computation of Time to Crash(Masters A. Negre)
• Natural Interest "Ridges" (Thesis Hai Tranh)
34
Scale and Orientation Normalised Gaussian RF's
i(i, j) Arg Max
{ 2G() A(i, j) }
Intrinisic Scale: Peak in Laplacian as a function of Scale.
Oriented Response can be obtained as a weighted sum of cardinal derivatives
<A(i,j) G> = <A(i,j) Gx> Cos() + <A(i,j) Gy > Sin()
Normalisation of scale and orientation provides invariance to distance and camera rotation.
35
Natural Interest Points(Scale Invariant "Salient" image features)
Local extrema of < G(i,j,)•A(i,j)> over i, j,
Problems with Points• Elongated shapes• Lack of discrimination power• No orientation information
Proposal: Natural Interest RidgesMaximal ridges in Laplacian Scale Space:
36
Natural Ridge Detection [Tran04]
Compute Derivatives at different Scales. For each point (x,y,scale)
Compute second derivatives: fxx,fyy,fxy Compute eigenvalues and eigenvectors of Hessian matrix Detect local extremum in the direction corresponding to the largest
eigenvalue. Assemble Ridge points,
2 f 2 fx2
2 fy2
Laplacian Hessian
37
QuickTime™ and a decompressor
are needed to see this picture.
38
Real Time, View Invariant Computer Vision
Current activity• Robust Visual Features for Face Detection • Direct Computation of Time to Crash• Natural Interest "Ridges" for perceptual organisation.
39
Research Problems
• Context-aware interactive environments • New forms of man-machine interaction (using
perception)• Real Time, View Invariant, Computer Vision• Autonomic Architectures for Multi-Modal Perception
40
Supervisor Provides:Execution Scheduler • Command InterpreterParameter Regulator • Description of State and Capabilities
Supervised Perceptual Process
Prediction
ObservationModulesObservationMo
dulesObservationModules
Es timation EntitiesVideo Stream
Time
Detection
ROI, S, Detection MethodROI, S, Detection Method
Autonomic Supervisor
Intepretation Actors
EventsCurrent StateResponse to commands
EventsConfiguration
Requests for state
41
Detection and Tracking of Entities
Entities: Correlated sets of blobsBlob Detectors: Backgrnd difference, motion,color, receptive fields histogramsEntity Grouper: Assigns roles to blobs as body, hands, face or eyes
QuickTime™ and a decompressor
are needed to see this picture.
42
Autonomic Properties provided by process supervisor
Auto-regulatory: The process controller can adapt parameters to maintain a desired process state.
Auto-descriptive: The process controller provides descriptions of the capabilities and the current state of the process.
Auto-critical: Process estimates confidence for all properties and events.
Self Monitoring: Maintaining a description of process state and quality of service
43
Self-monitoring Perceptual Process
• Process monitors likelihood of output• When an performance degrades, process adapts
processing (modules, parameters, and data)
PerceptualProcess
Error Classification
Error Recovery
ProcessModel
ModelLearning
UnknownErrors
Error?
Video
44
Autonomic Parameter Regulation
Parameter regulation provides robust adaptation to Changes in operating conditions.
Pixel-levelDetection
SystemParameters
VideoStream
TrackedEntities
EntitiesRecognition
ParameterRegulator
TrainingOperator Input
Operator
EntityDatabase
45
Research Contracts (2003-2006)
National and Industrial: ROBEA HR+ : Human-Robot Interaction (with LAAS and ICP)ROBEA ParkNav: Perception and action dynamic environmentsRNTL ContAct: Context Aware Perception (with XRCE)Contract HARP (Context aware Services - France Telecom)
IST - FP VI: Projet IST IP - CHIL : Multi-modal perception for Meeting Services
IST - FP V:Project IST - CAVIAR: Context Aware Vision for SurveillanceProject IST - FAME: Multi-modal perception for services Project IST - DETECT : Publicity Detection in Broadcast Video Project FET - DC GLOSS : Global Smart SpacesThematic Network: FGNet (« Face and Gesture »)Thematic Network: ECVision - Cognitive Vision
46
Collaborations
INRIA ProjectsEMOTION (INRIA RA): Vision for Autonomous Robots; ParkNav, ROBEA
(CNRS), Theses of C. Braillon and A. NegreORION (Sophia): Cognitive Vision (ECVision), Modeling Human Activity
Academic: IIHM, Laboratoire CLIPS: Human-Computer Interaction, Smart Spaces; Mapping
Project, IST Projects GLOSS, FAME, Thesis: J. LetissierUniv. of Karlsruhe (Multimodal interaction): IST FAME and CHIL.
IndustryFrance Telecom: (Lannion and Meylan) Project HARP, Thesis of M. Anne.Xerox Research Centre Europe: Project RNTL/Proact Cont'ActIBM Research (Prague,New York): Situation Modeling, Autonomic Software
Archictures, Projet CHIL
47
Knowledge Dissemination
61
02468
1012
2003 2004 2005 2006
Journal Conf & Wkshp ChaptersThesis Patents
48
Conferences and Workshops Organised
General Chairman (or co-chairman)Conference: SoC-EuSAI 2005 Workshops:Pointing 2004, PETS 2004, Harem 2005
Program Co-ChairmanInternational Conference on Vision Systems, ICVS 2003, European Symposium on Ambient Intelligence, EuSAI 2004,International Conference on Multimodal Interaction, ICMI 2005.
Program Committee/Reviewer: UBICOMP 2003, ScaleSpace 2003, sOc 03, ICIP 03, ICCV 03 AMFG 04, ICMI 03, RFIA 2004, IAS 2004, ECCV 2004,FG 2004, ICPR 2004, CVPR 2004, ICMI 2004, EUSAI 2004, CVPR 2005, ICRA 2005, IROS 2005, Interact 2005, ICCV05, ICVS 06, PETS 05, FG06, ECCV06, CVPR06, ICPR06, IROS06…
49
APP Registered Software
1) CAR : Robust Real-Time Detection and TrackingAPP IDDN.FR.001.350009.000.R.P.2002.0000.00000Commercial License to BlueEyeVideo
2) BrandDetect: Detection, tracking and recognition of commercial trademarks in broadcast video
APP IDDN.FR.450046.000.S.P.2003.000.21000Commercial License to HSArt
3) ImaLab: Vision Software Development Tool. Shareware, APP under preparation.Distributed to 11 Research Laboratories in 7 EU Countries
4) Robust Tracker v3.3 (stand-alone) 5) Robust Tracker v3.4 (Autonomic)6) Apte: Monitoring, regulation and repair of perceptual systems. 7) O3MiCID: Middleware for Intelligent Environments
50
Start-up: Blue Eye Video PDG: Pierre de la SalleMarketing : Jean ViscomteEngineers : Stephane Richetto
Pierre-Jean RiviereFabien PelissonDominique de Mont (HP)Sebastien Pesnel
Councelor : James L. Crowley
Incubation: INRIA Transfer, GRAIN, UJF Industrie. Region Rhône AlpesLauréat de concours création d'enterprise
Creation : 1 June 2003Market: Observation of human activity Sectors: Commercial services, Security, and traffic monitoring Status: 386 K Euros in sales in 2005, >100 Systems installed
5146
Blue Eye Video Activity Sensor(PETS 2002 Data)
QuickTime™ and a decompressor
are needed to see this picture.
52
Blue Eye Video Activity Sensor(Distributed Sensor Networks)
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
53
Evolutions for 2006-2010
Context-aware interactive environments• Adaptation and Development of Activity Models
New forms of man-machine interaction • Affective Interaction
Real Time, View Invariant, Computer Vision• Embedded View-invariant Visual Perception
Autonomic Architectures for Multi-Modal Perception• Learning for Monitoring and Regulation • Dynamic Service Composition
54
Automatic Adaptation and Development of Models for Human Activity
Adaptation: consistent behaviour across environmentsDevelopment: Acquisition of new abilities
Learning Situations
Situation Network
Learning (re)actions
Splitting Situations
Deleting obsolete Situations
Learning Roles
Supervisor Feedback
55
Affective interaction
Interactive objects that recognize interest and affect and that learn to perceive and evoke emotions in humans.
56
Embedded View-invariant Visual Perception
Embedded Real Time View Invariant Vision in phones and PDA’s (Work with ST MicroSystems)
57
Distributed Autonomic Systems for Multi-modal Perception
• Statistical Learning for Process Regulation and Monitoring• Dynamic Service Composition
User Services
Sensors, Actuators, CommunicationsLogical Sensors, Logical Actuators
Perceptual ComponentsSituation Modeling
Ontol ogy Serve r, Util iti es
58
PRIMAPerception Recognition and Integration
for Observing and Modeling Activity
James L. Crowley, Prof. I.N.P. GrenobleAugustin Lux, Prof. I.N.P. Grenoble
Patrick Reignier, MdC. Univ. Joseph FourierDominique Vaufreydaz, MdC UPMF
