1

Traitement, Modélisation d ’Images et Communications

(Processing, Modelling and Communication of images)

TEMICS

Evaluation 2001-2005

The Team and its Evolution

Nov. 2005

Project Leader C. Guillemot (INRIA)

Team Assistant H. Béchu (INRIA)

Researchers/Teachers T. Furon (INRIA) C. Fontaine (CNRS) L. Morin (Univ. Rennes-1) G. Rath (INRIA) + S. Marinkovic (INRIA), on leave, Sept.2004

Research Engineer L. Guillo (CNRS, 40%)

Temporary engineers: 3 Postdocs: 3 Ph.D Students: 9

Departures L. Amsaleg (creation of TexMex, 02) S. Pateux (secondment from FT, 03) P. Siohan (secondment from FT, 03) H. Nicolas (Univ Bordeaux 05)

Arrivals T. Furon (CR-INRIA, Oct. 02) L. Guillo (IG CNRS 40%, Oct. 02) S. Marinkovic (CR-INRIA, Oct. 02) G. Rath (CR-INRIA, Oct. 03) C. Fontaine (CR-CNRS, Oct. 05)

A. Roumy (CR-INRIA, Jan. 06)

Temporary engineers (2001-2005): 12 Postdocs (2001-2005): 11 Ph.D (2001-2005): 14

Changes

General Objectives

Design and development of theoretical frameworks, methods and algorithms in the areas of analysis, modelling, coding, communication and watermarking of images and video signals

1. Representation and compression with functionalities (navigation, manipulation, scalability, enabling feature extraction / indexing, …)

2. Communication over heterogeneous, wired & wireless networks

3. Information hiding under constraints of invisibility, capacity, robustness and security

1. Augmented reality, interaction / navigation with/in content

2. Networked (wired/wireless IP) multimedia

3. Copyright, content enrichment, copy protection, authentication

Application DomainsProblems

Research Axes and Methodology

Area 1: Video analysis and 3D modelling Computer vision Analysis-Synthesis based on meshes

Area 2: Scalable video representation Texture/geometry sparse representation Complete/over-complete transforms

Area 3: Joint source-channel coding Statistical and Turbo Estimation Joint Source-Channel Statistical Codes Frame Theory

Area 4: Watermarking and security Communication with side information Game theory Shannon cryptanalysis

Highlights of a few contributions

Challenges: Modeling of large scenes via a sequence of 3D models Automatic 3D modeling from unconstrained video Temporal consistency Multi-resolution low rate representation

Synthesis:• Temporal continuity of connectivity by construction,

• Temporal continuity of geometry via morphing

• Software transfer to France Telecom, contribution to MPEG/3D AV

Contribution: Analysis:

• Rate-distortion constrained construction of dense motion field + camera path depth map

• Consistent connectivity dissociated from geometry

• Synthesis wavelets on the low-res. mesh

Area 1 – Main Achievements: 3D Model-based scalable video coding

Contribution:

Local adaptation of orientations of 1D lifting steps on multi-resolution Quincunx grids Capture anisotropic geometrical structures

• Better energy compaction in low-frequencies. Outperforms in compression and denoising the other « *lets » (contourlets, curvelets, bandelets, …). Orientation optimization based on Markov fields in de-noising

Challenges: Compact (sparse) signal representation space Sparsity of the representation provided by WT

limited by the isotropy of the basis functions High energy coefficients cluster around edges.

Oriented Wavelet Transform - Anisotropic StructuresArea 2 – Main Achievements:

Challenges:

Sparse signal representation spaces Amenability to other processing tasks (motion analysis, feature extraction, local description)

Contributions:

Methods to tune the redundancy in Laplacian

Pyramid (LP) representations LP is a special type of frames

Reconstruction algorithms Foreseen Impact in the SVC context Patent filed, transfer under discussion.

Method-1

Method-2

Filters need not be bi-orthogonal or orthogonal

Area 2 – Main AchievementsBridging the gap between Wavelets and Laplacian Pyramids

Challenges: • High sensitivity to noise of VLCs • Optimal decoding in presence of noise • Quadratic complexity of optimal decoding solutions

Without errorsPSNR = 33.9 dB0.5 bpp

Hard dec. PSNR = 16.4 dB

Eb/N0=5 dB.

Soft dec PSNR = 25.1 dB

Contributions: • Estimation/soft synchro of VLC-coded sources

• Adopted in JPEG2K part 11

• Patent transferred/licensed to Thomson

• New State models based on state aggregation

• State model = (Nk,Tk mod T)• Complexity O(T.L) instead of O(L**2)• best paper award

Area 3 – Main Achievements: Statistical Estimation of VLC-encoded sources

Challenges: • Sub-optimality of classical VLCs in presence of noise

• Need for unequal error protection

• Sub-optimality when followed by a channel code

Contributions: • New Families of error-resilient source codes

• Multiplexed (stationary and 1st-order) codes: joint statistical coding of 2 sources• Reaching the source entropy• Synchronous decoding of high priority information• Patent transferred/licensed to Thomson

Area 3 – Main Achievements: New statistical codes for memoryless and memory sources

• Formalism based on re-writing rules extending the class of VLCs

leading to codes exhibiting a set of properties (compression, uniform marginal bit probability, reduced state space, …)

S l b

FLC

6.19 bpp

16.79 db

0.005 BER

Huf1.71 bpp15.59 db

0.005 BER

Mul

1.71 bpp

24.82 db

0.005 BER

Challenges: • Signal representation resilient to noise • Diversity in multi-channel transmission• Multiple description codes

Contributions: Signal expansion on redundant bases of functions (frames) A complete framework with a set of erasure recovery and error localization/recovery algos

• Frame theoretic (using the dual frame)

• Coding theoretic (syndrome) approaches

• Subspace-based approaches

• Hypothesis testing approaches

A large class of Frames (based on block codes, on filter banks, tree-structured wavelets)

IntroducingStructure in

signal

Sourcecoder channel

Sourcedecoder

ReconstructAlgorith

Frame = a bigger basis (overcomplete)

Redundancy structured by proper design

Syndromes can be expressed as a sum of complex sinusoids in noise.

9 Journal papers

Area 3 – Main Achievements: Over-Complete Frame Expansions and Decoding Methods

Contributions: Communication under constraints

Area 4 – Main Achievements: Robustness and Security of Watermarking

Challenges: Improving trade-off invisibility / capacity / robustness Assessing/improving watermarking security

8 bits hidden, 38 dB Scaling + JPEG (22dB)

Capacity: Channel coding with side information Robustness: Game theory (min-max game):

Security: Adaptation of Shannon cryptanalysis to watermarking• Measure information about secret key leaking from WM content• Theoretical analysis of security of main techniques (Fisher Information) (Number of observations / complexity needed) • Tools implementing the attack based on blind source separation

« Security » attack: 36 dB

xyxyb DD

N

E

iiiW

'0,

minmax

Best paper award

Patent

Visibility and Impact

14 Ph.D Theses 44 accepted journal publications + 6 submitted 7 book chapters 132 conference publications Thesis (2) and papers (2) awards

Organization of PCS 2003 Co-chairing CORESA 2005 Editorial board of IEEE Trans. on IP (00-04) Editorial board of IEEE Trans. on CSVT (04-06) TEMICS is present in International scientific

committees (IEEE-IMDSP, IEEE-MMSP, …) TEMICS is present in programme committees of a

large number of conferences (see synthesis doc.) Numerous expertises (see synthesis doc.)

5 INRIA patents

• 3 patents transferred/licensed to Thomson• Transfer of the 2 others under discussion

3 patents filed by industrial partners with

TEMICS members as co-authors

Software transfer

• 3D codec + 2D mesh-based codec transferred

to France Télécom (FT)

• WAVIX transferred to Thomson: Starting

point for joint reply to MPEG/SVC cfp.

• WULL provided to several companies

• MuxCode library transferred to Thomson

• AC estimation integrated in OpenJPEG and

transferred to Thomson

10 contributions to standardization

Scientific In the Application Domains2001-2005

Project Positioning within INRIA

Specificities w.r.t. teams in the area of 3D modelling (e.g., MOVI): unconstrained monocular sequences, constraints of description costs & scalability

Complementarity/collaboration with PLANETE on problems of cross-layer design: PLANETE focuses on MAC and network issues whereas TEMICS focuses on signal processing issues.

Complementarity/collaboration with TEXMEX on issues as tracing of copyrighted content: TEXMEX uses indexing techniques whereas TEMICS uses watermarking techniques; Study of transforms for joint compression - feature point / descriptor extraction

Strategic Objective:

« Develop multimedia data and multimedia information processing … To process multimedia data and transmitting it over heterogeneous networksThe objective of this research concerns modeling, processing and indexing large scale, changing data masses-signals, sounds, images, etc.-often associating several source modalities and several communication channel and data protection models. »

International / National Positioning

USA + Canada + Japan: • Berkeley (K. Ramchandran, A. Zhakor), Boston (J. Konrad), Rochester (M. Tekalp), Stanford (B. Girod), UIUC (P. Moulin, T. Huang), Tohoku Univ (K. Deguchi)

Europe: • Collaborations with UCL (B. Macq, L. Vandendorpe), EPFL (M. Kunt), Vigo (F. Perez- Gonzales), Sienna (M. Barni), Geneva (T. Pun), … + many others via 6 EU projects.

France: • Collaborations with CNRS-L2S, I3S, LIS-Grenoble, ENST-Paris

TEMICS strengths:• Problematic at the crossroad of information theory, signal processing, and vision

• Research spectrum capitalizing on dualities between a range of problems

• Scientific (publications) and applicative (patents,standardization) achievements

Peer / Competing Groups

Summary

Recommendations of last evaluation• Focus on telecommunication problems (« on the hottest research areas of visual communication»)

• Contribute to standardization

• Patents and software transfer

• Re-enforce the team on the main focus of the project

High visibility and high impact on scientific and applicative aspects

The team forms a very good mix of signal processing, information theory and vision skills needed for the problems addressed at the crossroad of the three domains.

2001 - 2005

Smooth continuation and evolution of the most promising research directions as well as new challenging research directions to address the three following problems:

1. Representation and compression with functionalities (navigation, manipulation, scalability, enabling feature extraction / indexing, …)

2. Communication over heterogeneous, wired & wireless networks

3. Information hiding under constraints of invisibility, capacity, robustness, security

Objectives for next period

5 Research Axes

Area 1: Video analysis and 3D modelling

Area 2: Scalable video representation

Area 3: Joint source-channel coding

Area 4: Watermarking and security

Area 5: Distributed Source Coding

New Challenges (1/2)

Area 1: Video Analysis and 3D Modelling Vision for compression and communication Higher level/multimodal information (e.g., GIS data) with a range of fundamental and methodological issues (registration, extraction of correlated features, joint analysis and representation)

Area 2: Scalable Representation/Coding Multi-resolution and sparse representations: trade-offs sparsity, decay order, invariance, coherence across scales, space and directions, frequency selectivity, …. Multi-resolution spatio-temporal texture analysis/synthesis

Area 3: Joint Source-Channel Coding Signal processing (e.g., frames, framelets) and information theoretic (statistical codes, JSC codes, iterative/turbo decoding) approaches for video communication (including multiple channels)

Area 4: Watermarking and Security Extend our work on watermarking security assessment considering other hiding primitives (authentication, fingerprinting, steganography) Address the poor trade-off robustness/security of watermarking techniques: Is there a fundamental reason (theoretical bounds) for the poor trade-off robustness/security ?

New Challenges (2/2)

Area 5: Distributed Source Coding (New, started in 2004) A large range of open fundamental and applied problems

• Side information modelling (3D for exploiting scene geometrical constraints)

• Code design accounting for source memory (Frame theory, FSM) and channels

Thanks for listening!

Questions ?