1

RUSProjects Communication Systems

BeWü Development

RTP Payload for MPEG-4 Payload for MPEG-4

with Flexible Error Resiliencywith Flexible Error Resiliency draft-ietf-avt-mpeg4streams-00.txt

proposed experimental RFC

formerly draft-guillemot-avt-genrtp-03.txt

RTP Payload for MPEG-4 Payload for MPEG-4

with Flexible Error Resiliencywith Flexible Error Resiliency draft-ietf-avt-mpeg4streams-00.txt

proposed experimental RFC

formerly draft-guillemot-avt-genrtp-03.txt Christine Guillemot, Paul Christ, Stefan Wesner [, Anders Klemets]

2

RUSProjects Communication Systems

BeWü Development

draft-guillemot-avt-genrtp-03.txt => new title Flexmux section added in discussion with authors - (see draft-

rgcc-flexmuxmpeg4-00.txt);

Payload Type (PT): Different payload types should be assigned for MPEG4 ES, MPEG4 SL-PDU MPEG-4 FlexMux streams. A payload type in the dynamic range should be chosen.

Same format, same interface ES, SL, Flexmux

TSOFFSET removed, Ebits included

draft-guillemot-avt-genrtp-03.txt => new title Flexmux section added in discussion with authors - (see draft-

rgcc-flexmuxmpeg4-00.txt);

Payload Type (PT): Different payload types should be assigned for MPEG4 ES, MPEG4 SL-PDU MPEG-4 FlexMux streams. A payload type in the dynamic range should be chosen.

Same format, same interface ES, SL, Flexmux

TSOFFSET removed, Ebits included

Modifications since WashingtonModifications since Washington

3

RUSProjects Communication Systems

BeWü Development

Multiple Dimensions of MPEG-4/RTPMultiple Dimensions of MPEG-4/RTP

Systems (OD) or non-systems (non-OD) framework

Various types of streams with different QoS requirements (MPEG-4 A/V, OD, BIFS, IPMP, other compressed AV

formats, eg. H.263 streams)

Mixed live and streamed (pre-encoded) content

Session Types Client - Server (unicast), “MU” (Multi) Peer-to-Peer - (multicast)

Systems (OD) or non-systems (non-OD) framework

Various types of streams with different QoS requirements (MPEG-4 A/V, OD, BIFS, IPMP, other compressed AV

formats, eg. H.263 streams)

Mixed live and streamed (pre-encoded) content

Session Types Client - Server (unicast), “MU” (Multi) Peer-to-Peer - (multicast)

... in applications relying on

4

RUSProjects Communication Systems

BeWü Development

Unified solution for the transport of MPEG-4 MPEG-4 SL packet streams - and for MPEG-4 ES

Common solution for both live and pre- encoded content

Applicable to the transport of FlexMux PD (submitted draft-rgcc-avt-mpeg4flexmux-00.txt)

Unified solution for the transport of MPEG-4 MPEG-4 SL packet streams - and for MPEG-4 ES

Common solution for both live and pre- encoded content

Applicable to the transport of FlexMux PD (submitted draft-rgcc-avt-mpeg4flexmux-00.txt)

Summary Summary

5

RUSProjects Communication Systems

BeWü Development

SummarySummary Abstraction of

QoS monitoring and Congestion Contro “intelligence” unique interface architectural simplicity and consistent stream “adaptation” in high number of streams applications (mixing real-time & pre-encoded content)

Packet Loss flexible protection mechanisms Full and partial AUs Segments with types or priority

Abstract media idiosyncrasies Assuming a media and network aware adaptation layer Towards UEP and/or Diffserv marking based on AUs

Abstraction of QoS monitoring and Congestion Contro “intelligence” unique interface architectural simplicity and consistent stream “adaptation” in high number of streams applications (mixing real-time & pre-encoded content)

Packet Loss flexible protection mechanisms Full and partial AUs Segments with types or priority

Abstract media idiosyncrasies Assuming a media and network aware adaptation layer Towards UEP and/or Diffserv marking based on AUs

6

RUSProjects Communication Systems

BeWü Development

Sample RTP packet for MPEG-4 ESsSample RTP packet for MPEG-4 ESs

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0|0| Res | | +-+-+-+-+-+-+-+-+ | | . . AU or partial AU . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

If no additional data-based protection needed

- One byte difference with draft-ietf-avt-mpeg4-es-00.txt,Byte offering additional flexibility in terms of protection- The packet would be of this form if protection supported by compression layer in the case of real-time

7

RUSProjects Communication Systems

BeWü Development

Sample RTP packet for MPEG-4 ESsSample RTP packet for MPEG-4 ESs

If additional data-based protection needed (eg. duplicatedheaders), (no fragmentation used).

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |G|E| XT | LENGTH |EBITS| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | . + Extension data (e.g. VOP header) +-+-+-+-+-+-+-+-+ . |G|E|0| res | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | . . Media Payload | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

8

RUSProjects Communication Systems

BeWü Development

Sample RTP packet for MPEG-4 SL-PDUsSample RTP packet for MPEG-4 SL-PDUs

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |G|E|F| res | optional SL header paramaters as indicated by . +-+-+-+-+-+-+-+-+ the SLConfigDescriptor . | . . Media payload | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

If no additional data-based protection needed

9

RUSProjects Communication Systems

BeWü Development

Sample RTP packet for MPEG-4 SL-PDUsSample RTP packet for MPEG-4 SL-PDUs

If additional protection needed

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |G|E| XT | LENGTH |EBITS| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | . . Extension data . | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |G|E|F| res | optional SL header paramaters as indicated by | +-+-+-+-+-+-+-+-+ the SLConfigDescriptor . | . . Media payload | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

10

RUSProjects Communication Systems

BeWü Development

Usage of the format for MPEG-4 Flexmux-PDUs in draft-rgcc-avt-mpeg4flexmux-00.txt

Usage of the format for MPEG-4 Flexmux-PDUs in draft-rgcc-avt-mpeg4flexmux-00.txt« What is suggested here is to define objects in the RTP payload, asit is proposed in draft-ietf-avt-mpeg4streams-01.txt. The RTP payload being a succession of ob-jects. Each object is byte aligned. Each object starting with its length. »

« A payload object is either a protection data object, or a completeFlexMux packet. Each payload object follows an identification byte,its object type byte. The RTP payload starts with one object typebyte. »

« The object type byte syntax: 0 1 2 3 4 5 6 7 +-+-+-+-+-+-+-+ |G|E| XT | +-+-+-+-+-+-+-+ »

11

RUSProjects Communication Systems

BeWü Development

MPEG-4/RTP recent historyMPEG-4/RTP recent history 12/98 Orlando,

draft-ietf-avt-rtp-mpeg4-00.txt, SL-packetized streams draft-guillemot-avt-genrtp-00.txt, ES streams + flexible additional

protection (if needed, eg. for streamed content)

04/99 NY + 07/99 Oslo joint IETF/MPEG (phone) meetings “No non-SL- and non-Systems ES RTP-mapping needed”: “2 experimental RFCs ...”

10/99 ISO Melbourne JNB non-System AV ES-mapping proposed

12/99 Washington draft-jnb-mpeg4av-rtp-00.txt „the normative way on how MPEG-4 Audio/Visual streams ... mapped ... to RTP“

12/98 Orlando, draft-ietf-avt-rtp-mpeg4-00.txt, SL-packetized streams draft-guillemot-avt-genrtp-00.txt, ES streams + flexible additional

protection (if needed, eg. for streamed content)

04/99 NY + 07/99 Oslo joint IETF/MPEG (phone) meetings “No non-SL- and non-Systems ES RTP-mapping needed”: “2 experimental RFCs ...”

10/99 ISO Melbourne JNB non-System AV ES-mapping proposed

12/99 Washington draft-jnb-mpeg4av-rtp-00.txt „the normative way on how MPEG-4 Audio/Visual streams ... mapped ... to RTP“

12

RUSProjects Communication Systems

BeWü Development

MPEG-4/RTP recent historyMPEG-4/RTP recent history 03/2000 Adelaide,

draft-rgcc-avt-flexmuxmpeg4-00.txt, flexmux-packetized streams -

– withdrawn last week in the Netherlands ??

.... more to come?

03/2000 Adelaide,

draft-rgcc-avt-flexmuxmpeg4-00.txt, flexmux-packetized streams -

– withdrawn last week in the Netherlands ??

.... more to come?

13

RUSProjects Communication Systems

BeWü Development

Implementation statusImplementation status

Mapping/de-mapping completed Integrating into IM1 as a DMIF plug-in (with

CSELT) is ongoing FEC implemented: duplicated headers, parity codes Adaptivity for QoS monitoring under development R-S under development

Mapping/de-mapping completed Integrating into IM1 as a DMIF plug-in (with

CSELT) is ongoing FEC implemented: duplicated headers, parity codes Adaptivity for QoS monitoring under development R-S under development

14

RUSProjects Communication Systems

BeWü Development

Information - DistributionInformation - Distribution current code documentation

http://www-ks.rus.uni-stuttgart.de/PROJ/GP ACTS COMIQS project ended february 2000

http://www.ccett.fr/comiqs/welcome.htm Used in 1 National Project (F, RNRT-VISI) Continue in 2 European Projects (IST-SONG, IST-

OPENISE) Distributed to 4 Companies outside the above

projects

current code documentation http://www-ks.rus.uni-stuttgart.de/PROJ/GP

ACTS COMIQS project ended february 2000 http://www.ccett.fr/comiqs/welcome.htm

Used in 1 National Project (F, RNRT-VISI) Continue in 2 European Projects (IST-SONG, IST-

OPENISE) Distributed to 4 Companies outside the above

projects

15

RUSProjects Communication Systems

BeWü Development

ConclusionConclusion Complexity of (future) MPEG-4 usage

Multiple streams (live and pre-encoded) with different QoS requirements so far, multiple ways for handling protection in the different ESs (HEC specific to visual)

Stresses the need, for the sake of architectural « simplicity »,

simple, single interface, avoiding parsers of the ESs

draft-ietf-avt-mpeg4streams-01.txt has the potential for an homogeneous way of handling efficient transport of the different types of MPEG-4 streams

Complexity of (future) MPEG-4 usage Multiple streams (live and pre-encoded) with different QoS requirements so far, multiple ways for handling protection in the different ESs (HEC specific to visual)

Stresses the need, for the sake of architectural « simplicity »,

simple, single interface, avoiding parsers of the ESs

draft-ietf-avt-mpeg4streams-01.txt has the potential for an homogeneous way of handling efficient transport of the different types of MPEG-4 streams

16

RUSProjects Communication Systems

BeWü Development

Application Case of

draft-ietf-avt-mpeg4streams-01.txt

to MPEG-4 Visual streams

in draft-gc-avt-mpeg4visual-00.txt

Application Case of

draft-ietf-avt-mpeg4streams-01.txt

to MPEG-4 Visual streams

in draft-gc-avt-mpeg4visual-00.txt

Christine Guillemot, Paul Christ, Stefan Wesner

17

RUSProjects Communication Systems

BeWü Development

Application of drat-ietf-avt-mpeg4streams-01.txt to the transport of MPEG-4 visual streams

Common solution for both live and pre- encoded content

Common solution for the different profiles

Application of drat-ietf-avt-mpeg4streams-01.txt to the transport of MPEG-4 visual streams

Common solution for both live and pre- encoded content

Common solution for the different profiles

Summary Summary

18

RUSProjects Communication Systems

BeWü Development

Simple Visual Profile: efficient, error resilient coding of rectangular

video objects, suitable for applications on mobile networks.

Simple Scalable Profile: adds support for coding of temporal and spatial scalable objects, useful for applications which provide services at more than one level of quality due to bit-rate or decoder resource limitations.

Core Visual Profile: adds support for coding of arbitrary-shaped and temporally scalable objects, useful for applications such as those providing content-interactivity (Internet multimedia applications).

Simple Visual Profile: efficient, error resilient coding of rectangular

video objects, suitable for applications on mobile networks.

Simple Scalable Profile: adds support for coding of temporal and spatial scalable objects, useful for applications which provide services at more than one level of quality due to bit-rate or decoder resource limitations.

Core Visual Profile: adds support for coding of arbitrary-shaped and temporally scalable objects, useful for applications such as those providing content-interactivity (Internet multimedia applications).

Visual Profiles Visual Profiles

19

RUSProjects Communication Systems

BeWü Development

Main Visual Profile: adds support for coding of interlaced, semi-

transparent, and sprite objects to the Core Visual Profile; useful for

interactive and entertainment-quality broadcast and DVD applications.

N-Bit Visual Profile: adds support for coding video objects having

pixel-depths ranging from 4 to 12 bits to the Core Visual Profile.

…. More for synthetic and for natural/synthetic hybrid visual content

Main Visual Profile: adds support for coding of interlaced, semi-

transparent, and sprite objects to the Core Visual Profile; useful for

interactive and entertainment-quality broadcast and DVD applications.

N-Bit Visual Profile: adds support for coding video objects having

pixel-depths ranging from 4 to 12 bits to the Core Visual Profile.

…. More for synthetic and for natural/synthetic hybrid visual content

Visual Profiles (continued) Visual Profiles (continued)

20

RUSProjects Communication Systems

BeWü Development

Very elaborate and powerfull syntax.

HEC mechanism useable initially for protecting header parameters needed in the simple profile

Lately corrigendum (DCOR1) in order to cover header parameters for arbitrary shape objects

Not (yet?) for enhancement layers when scalability is used in simple scalable, core, main

Very elaborate and powerfull syntax.

HEC mechanism useable initially for protecting header parameters needed in the simple profile

Lately corrigendum (DCOR1) in order to cover header parameters for arbitrary shape objects

Not (yet?) for enhancement layers when scalability is used in simple scalable, core, main

Impact of Profiles on SyntaxImpact of Profiles on Syntax

21

RUSProjects Communication Systems

BeWü Development

Configuration information global configuration information refering to the whole group

of visual objects (visualobjectsequence()), object configuration information refering to a single visual

object (visualobject()) object layer configuration information (visualobjectlayer()) video object plane header (videoObjectPlane()) (for video)

Configuration information global configuration information refering to the whole group

of visual objects (visualobjectsequence()), object configuration information refering to a single visual

object (visualobject()) object layer configuration information (visualobjectlayer()) video object plane header (videoObjectPlane()) (for video)

Elements of syntaxElements of syntax

22

RUSProjects Communication Systems

BeWü Development

Two modes of transmission for visual object

sequence layer, visual object layer and video

object layer: separate mode (in containers provided by MPEG-4 system (Ods)

combined mode (as part of the ES)

video object plane header as part of ES

Two modes of transmission for visual object

sequence layer, visual object layer and video

object layer: separate mode (in containers provided by MPEG-4 system (Ods)

combined mode (as part of the ES)

video object plane header as part of ES

Transmission of key segmentsTransmission of key segments

23

RUSProjects Communication Systems

BeWü Development

To avoid to have to support parsers in streaming applications

Same way of handling protection for real-time and pre-encoded content

simple and same interface Esdata dataLength number of padding bits at the end (if needed) degradationPriority and/or segment type

(videoobjectlayer(), GOV header, video object plane header, video packet)

To avoid to have to support parsers in streaming applications

Same way of handling protection for real-time and pre-encoded content

simple and same interface Esdata dataLength number of padding bits at the end (if needed) degradationPriority and/or segment type

(videoobjectlayer(), GOV header, video object plane header, video packet)

Uniform way of handling protection of key segmentsUniform way of handling protection of key segments

24

RUSProjects Communication Systems

BeWü Development

Sample RTP packet for MPEG-4 visualstreamsSample RTP packet for MPEG-4 visualstreams

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |G|E| XT | LENGTH |EBITS| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | . + eg. Video Object Plane header +-+-+-+-+-+-+-+-+ . |G|E|0| res | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | . . Media Payload | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+