E.G.M. Petrakis The MPEG Standard 1

The MPEG Standard

MPEG-1 (1992) actually a video player plays out audio/video streamssame type of access as home VCR

MPEG-2 (1995) introduced for compression and transmission of digital TV signalsstill limited interactivity

MPEG-4 (1999) is completely different high level of interactivity

MPEG-7 (2002) for the description of metadata only

E.G.M. Petrakis The MPEG Standard 2

MPEG-4

MPEG-4 addresses the need towardsMixing of natural and synthetic audiovisual

informationHigh interactivity in the presentation of

multimedia contentDeployment of communication systems for real-

time or broadcast delivery of coded data streamsA new approach for describing, coding and

presenting a sceneMPEG-4 combines different coding tools for

Audio/video Synthetic objects and graphics

E.G.M. Petrakis The MPEG Standard 3

MPEG-4 Objects

The audio/video components of MPEG-4Objects are coded, transmitted separately

and composed at the decoder siteThey can exist independentlyMultiple objects can be grouped together to

form complex objectsVideo and audio can be easily manipulatedPermits choosing appropriate coding tools

for audio, video and graphics objects

E.G.M. Petrakis The MPEG Standard 4

MPEG-4 Object Based Coding

E.G.M. Petrakis The MPEG Standard 5

MPEG-4 Coding

The scene is composed and rendered at the sender sitevideo frames, audio are coded, multiplexed

and transmitted tools for coding arbitrarily shaped objects

At the receiver the stream is demultiplexedvideo and audio are decoded, composed,

synchronized and presented as defined at the senders site

E.G.M. Petrakis The MPEG Standard 6

Object Coding

Objects are described mathematically (e.g. by their positions)similarly for audio and graphics objectsan object need only be defined oncethe viewer can change their positiontransmit calculations to update the scene

at the receiverthis is a critical feature when the response

has to be fast and bit-rate is limited

E.G.M. Petrakis The MPEG Standard 7

Binary Format for Scenes (BIFS)

MPEG-4’s language for describing and dynamically changing a scene

Borrows concepts from VRMLBoth define representations of the same dataVRML defines objects and actions in textBIFS code is binary (10-15 times shorter)Unlike VRML, MPEG-4 uses BIFS for real-time

streaming: a scene can be built-up and played on the fly

VRML and BIFS evolve consistently

E.G.M. Petrakis The MPEG Standard 8

scene graph

E.G.M. Petrakis The MPEG Standard 9

The Scene Graph

Represents a scene as independent or compound objects e.g., father and child the audio track of his voicefloor and walls (sprites: for

backgrounds)the web site the synthetic image of the furniturea synthetic HDTV set playing a movie

from the families DVD library

E.G.M. Petrakis The MPEG Standard 10

Elementary Streams (ES)The scheme for preparing content for

transmission, storage and decodingObjects are placed in ESsProbably two or more ESs per objectA sound track or a video may have a single

ESScalable objects way have one ES for basic

quality information + one or more enhancement layers for improved quality (e.g., finer detail, faster motion)

ESs are split into packets and sent along with timing information for proper synchronization

E.G.M. Petrakis The MPEG Standard 11

Object Descriptors (OD)

MPEG-4s mechanism that informs the system which ES belongs to a certain objectOD contain Elementary Stream

Descriptors (ESD) which tell the system which decoders to use

ODs are sent in their own stream which allows them to be added or deleted as the scene changes

E.G.M. Petrakis The MPEG Standard 12

Profiles and Levels

MPEG-4 provides a set of tools for coding multimedia contentsan application may use only subsets of these

toolsProfiles: MPEG-4s definitions of these

subsets for audio, visual, graphics information

Levels: define the computational complexity of the profile’s tool subset

Certain combinations of profiles fit well together

E.G.M. Petrakis The MPEG Standard 13

MPEG-4 Profiles

E.G.M. Petrakis The MPEG Standard 14

MPEG-4 Visual Objects

Arbitrarily shaped objects are coded apart from their background

Binary shape coding: a pixel is or is not part of an object simple, crude technique, suitable for low-bit

rates, suffers from aliasing

Alpha shape (gray scale) coding: each pixel is assigned a value for its transparencyobjects can be smoothly blended into a

background or with other objects

E.G.M. Petrakis The MPEG Standard 15

Visual Objects

Rectangular natural images and scenes are coded using MPEG-1, 2

Texture is coded separately by a DCT, block based coding scheme or wavelets

E.g., weather reports: the weatherman’s image seems to be standing in front of a map which is actually generated elsewhere

E.G.M. Petrakis The MPEG Standard 16

Object Segmentation

MPEG does not specify how objects are extracted video object segmentation is difficult e.g., record weatherman’s image in

front of a color background MPEG-4 specifies decoding

implementation of encoding is left to the industry to decide

E.G.M. Petrakis The MPEG Standard 17

MPEG-4 Applications

MPEG-4 makes video possible even at very low bit-rates (e.g., 10 kb/s)mobile devices, internet

Scalable objects for low bit-ratesa base layer conveys all the information

in some basic quality one of more enhancement layers can be

sent to get better qualitysend only the most important objects

E.G.M. Petrakis The MPEG Standard 18

Sprites

For coding unchanged backgrounds The background is defined and coded

only onceMust be updated for each change

(e.g., when the viewing angles changes)

The sprite is sent only onceNew views are created by sending the

new positions

E.G.M. Petrakis The MPEG Standard 19

Advanced Features

Map images into computer generated shapesa 2D or 3D mesh may have an image mapped onto ita few parameters to deform the mesh generate the

impression of a moving picture rather than sending new images for each change,

send commands and parameters to the viewerpre-defined faces are particularly interesting meshesthe appearance of a face may be left to the decoder

(e.g., custom facial models can be downloaded)

E.G.M. Petrakis The MPEG Standard 20

MPEG-4 Faces

Images laid over a wire-frame face

Send wire-frame plus parameters

Image reconstruction at receivers site

Speech is generated from text in steps with motions of the mouth, eyes and lips

E.G.M. Petrakis The MPEG Standard 21

MPEG-7MPEG-7 (2002) focuses on description of

multimedia contentmodalities: image, speech, video, graphics and

their combinationsMPEG-7 complements existing MPEG

standards and is applicable even to non-MPEG formats (compressed or uncompressed)

MPEG-7 is driven by trends in technology, market and user needs

Applications: VideoOnDemand, NewsOnDemand, InteractiveTV, multimedia information systems etc.

E.G.M. Petrakis The MPEG Standard 22

Scope of the StandardProvides the means for indexing,

searching, filtering and managing audio-visual contentbroadcast media selection (e.g.,

personalized TV)multimedia editing (e.g., personalized

news service)MPEG-7 interoperable interface

defines syntax and semantics tools may be designed for specific

modalities, aspects or applications

E.G.M. Petrakis The MPEG Standard 23

Interoperable Servicesand Applications

E.G.M. Petrakis The MPEG Standard 24

MPEG-7 Main Tasks

Multimedia: generate customized program guides or summaries of broadcast audio-visual content

Archive: generate descriptions of audio-visual content (or elements)

Adaptation: filter and transform multimedia streams in low bit-rate environments (e.g., mobile users)

E.G.M. Petrakis The MPEG Standard 25

MPEG-7 Specific Tasks

Music/audio: play a few notes and return music with similar music/audio

Images/graphics: draw a sketch and return images with similar graphics

Movement: describe movements and return video clips with the specified temporal and spatial relations

Scenario: describe actions and return scenarios where similar actions take place

E.G.M. Petrakis The MPEG Standard 26

MPEG-7 Elements

1. Descriptors (D) : define syntax and semantics of features of audio-visual content

Application independent Low level: shape, motion, color,

camera motion, harmonicity, timbre for audio ...

Semantic level: events, concepts ...

E.G.M. Petrakis The MPEG Standard 27

MPEG-7 Elements (cont.d)

2. Description Schemes (DS): specify the structure and semantics of the relationships among the constituent Ds or DSs e.g.,

Video DS specify syntax and semantics for segment decomposition, attributes, their relationships

DS related to creation, production, and access of content (e.g., property rights, parental rating, etc.)

E.G.M. Petrakis The MPEG Standard 28

MPEG-7 Elements (cont.d)

3. Description Definition Language (DDL): allows flexible definition of Ds and DSs based on XML schema

Ds and DSs are application independent

DLLs to define specialized tools

E.G.M. Petrakis The MPEG Standard 29

MPEG-7 Descriptions

MPEG-7 allows descriptions at different levels of abstractionslow level features extracted automaticallysemantic features with human interaction

or textual annotationMPEG-7 does not specify how features

are extracted or used (e.g., filtering, retrieval)their representation must conform to the

MPEG-7 standard

E.G.M. Petrakis The MPEG Standard 30

MPEG-7 Parts

Systems: specifies functionality at system level Preparation of descriptions for efficient

transport and storage synchronization of content and descriptorsdevelopment of decoders

Description Definition Language (DDL): language for specifying new Ds and DSsextension of XML schema

E.G.M. Petrakis The MPEG Standard 31

MPEG-7 Visual

Specifies a set of standardized visual Ds and DSsColor descriptors: color space, quantizationTexture descriptors: homogeneous texture,

texture browsing, edge histogram ...Shape descriptors: for regions or contoursMotion descriptors: camera motion,

trajectories, motion activity ...Face recognition

E.G.M. Petrakis The MPEG Standard 32

MPEG-7 Audio

Specifies standardized audio descriptors and descriptor schemes for pure music, pure speech, sound effects, soundtrackssilence descriptor spoken content descriptors sound effects descriptorsmelody contour descriptors

E.G.M. Petrakis The MPEG Standard 33

Multimedia Description Schemes

Specify a framework that allows generic description of all kinds of multimedia data

basic elements: data types, structures, Ds content management: content from

several viewpoints (creation, usage etc.) organization of content by collections,

classification navigation and access user interaction

E.G.M. Petrakis The MPEG Standard 34

Multimedia Description Schemes

E.G.M. Petrakis The MPEG Standard 35

MPEG-7 Reference Software

Reference implementation of the relevant parts of the MPEG-7 standardThe focus is on creating bit-streams of

descriptors and description schemes (DDL parser, DDL validation, multimedia description schemes)

Some software for extracting descriptors is also included (visual, audio descriptors)

E.G.M. Petrakis The MPEG Standard 36

References1. “MPEG-4 Multimedia for our Time” R. Koenen,

IEEE Spectrum, Feb. 1999, pp. 26-332. “

Applying and Implementing the MPEG-4 Multimedia Standard”, J. Kneip et.al. IEEE Micro, Nov-Dec 1999, pp. 64-74

3. “Overview of the MPEG-7 Standard”, S.-Fu Chang, T. Sikora and A. Puri, IEEE Transactions on Circuits and Systems for Video Technology, special issue on MPEG-7, June 2001

4. “Everything You Wanted to Know about MPEG-7” F. Nack and A.T. Lindsay, Part I, II, IEEE Multimedia, Aug-Dec1999