05. May 2004 1INF-3190: Multimedia
Protocols
Multimedia Protocols
Foreleser: Carsten GriwodzEmail: [email protected]
05. May 2004 2INF-3190: Multimedia
Protocols
Non-QoSMultimedia Networking
RTP – Real-Time Transfer Protocol
05. May 2004 3INF-3190: Multimedia
Protocols
Real-time Transport Protocol (RTP)
Real-time Transport Protocol (RTP) RFC 1889 Designed for requirements of real-time data transport NOT real-time NOT a transport protocol
Two Components Real-Time Transfer Protocol (RTP) RTP Control Protocol (RTCP)
Provides end-to-end transport functions Scalable in multicast scenarios Media independent Mixer and translator support RTCP for QoS feedback and session information
05. May 2004 4INF-3190: Multimedia
Protocols
Real-time Transport Protocol (RTP)
No premise on underlying resources
layered above transport protocol
no reservation / guarantees
Integrated with applications
RTP follows principles of Application Level Framing
and Integrated Layer
Processing
UDP
IPv4/6
EthernetAAL5ATM
ST-2
RTP RTCP
mediaencapsulation
application
TCP
ATM
05. May 2004 5INF-3190: Multimedia
Protocols
RTP RTP services are
Sequencing Synchronization Payload identification QoS feedback and session information
RTP supports Multicast in a scalable way Generic real-time media and changing codecs on the fly Mixers and translators to adapt to bandwidth limitations Encryption
RTP is not designed for Reliable delivery QoS provision or reservation
05. May 2004 6INF-3190: Multimedia
Protocols
RTP Functions RTP with RTCP provides
Support for transmission of real-time data Over multicast or unicast network services
Functional basis for this Loss detection – sequence numbering Determination of media encoding Synchronization – timing Framing - “guidelines” in payload format definitions Encryption Unicast and multicast support Support for stream “translation” and “mixing” (SSRC;
CSRC)
05. May 2004 7INF-3190: Multimedia
Protocols
RTP Packet FormatTypical IETF RFC bit-exact representation
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P|X| CC |M| PT | SEQ | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | TST | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | synchronization source (SSRC) identifier | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | contributing source (CSRC) identifiers | | .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| header extension |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| payload (audio, video, ...) || .... |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
a byte
a longword (32 bit)
05. May 2004 8INF-3190: Multimedia
Protocols
RTP Packet Format
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P|X| CC |M| PT | SEQ | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | TST | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | synchronization source (SSRC) identifier | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | contributing source (CSRC) identifiers | | .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| header extension |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| payload (audio, video, ...) || .... |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4 bit CSRC count, indicates the number of contributing sources in the header
Marker bitMeaning depends on payload
profile,e.g. frame boundary
Version number,Always 2
Padding indicator bitif set, number of padding bytes is in last byte of payload
Header extension bitTrue if header extension is present
7 bit payload typeAllows identification of thepayload’s content type
05. May 2004 9INF-3190: Multimedia
Protocols
RTP Packet Format
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P|X| CC |M| PT | SEQ | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | TST | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | synchronization source (SSRC) identifier | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | contributing source (CSRC) identifiers | | .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| header extension |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| payload (audio, video, ...) || .... |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 bit sequence number
Several 32 bit CSRCContribution source identifier, the number is
indicated by CCA mixer copies the original sources’ SSRCs
here
32 bit timestamp
32 bit SSRCSynchronization source identifier, a random
number identifying the sender
Header extensionmultiples of 32 bit
05. May 2004 10INF-3190: Multimedia
Protocols
RTP Architecture Concepts Integrated Layer Processing
Typical layered Data units sequentially processed by each layer
Integrated layer processing Adjacent layers tightly coupled
Therefore, RTP is not complete by itself: requires application-layer functionality/information in header
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P|X| CC |M| PT | SEQ | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | TST | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | synchronization source (SSRC) identifier | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | contributing source (CSRC) identifiers | | .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| header extension |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| payload (audio, video, ...) || .... |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 bit sequence number32 bit timestamp
7 bit payload typeAllows identification of thepayload’s content type
Marker bitMeaning depends on payload profile, e.g. frame boundary
05. May 2004 11INF-3190: Multimedia
Protocols
RTP Packet Format Relatively long header (>40 bytes)
overhead carrying possibly small payload header compression other means to reduce bandwidth (e.g. silence
suppression)
No length field Exactly one RTP packet carried in UDP packet Can use TCP or ATM AAL5
do-it-yourself packaging
Header extensions for payload specific fields possible
Specific codecs Error recovery mechanisms
05. May 2004 12INF-3190: Multimedia
Protocols
RTP Profile (RFC 1890) Set of standard encodings and payload types
Audio: e.g. PCM-u, GSM, G.721 Video: e.g. JPEG, H.261
Number of samples or frames in RTP packet Sample-based audio: no limit on number of samples Frame-based audio: several frames in RTP packet
allowed
Clock rate for timestamp Packetized audio: default packetization interval 20 ms Video: normally 90 kHz, other rates possible
05. May 2004 13INF-3190: Multimedia
Protocols
RTP Profiles Payload type identification
RTP provides services needed for generic A/V transport Particular codecs with additional requirements Payload formats defined for each codec: syntax and semantic of
RTP payload Payload types
Static: RTP AV profile document Dynamic: agreement on per-session basis
Profiles and Payload Formats in RTP Framework
RTP / RTCP
AV Profile
AdditionalProfiles
PayloadFormats
Dynamic Payload Types
PT mapping outside RTP
(e.g. SDP)
05. May 2004 14INF-3190: Multimedia
Protocols
RTP Profiles General
Associated with a media type. Provides association between PT field and specific media
format Defines sampling rate of timestamp May also define or recommend a definition for the
“marker” bit
Video Profile Marker bit recommended to mean last packet
associated with a timestamp Timestamp clock: 90000 Hz Defines PT mapping for a number of different video
encoding types
05. May 2004 15INF-3190: Multimedia
Protocols
RTP Profiles Audio Profile
Marker bit set on the first packet after a silence period where no packets sent
Timestamp equals sampling rate Recommends 20ms minimum frame time Recommends that samples from multiple channels be
sent together Defines PT for a number of different audio encoding
types
05. May 2004 16INF-3190: Multimedia
Protocols
RTP Profile for MPEG Video Payload
Frame headersGOP header
05. May 2004 17INF-3190: Multimedia
Protocols
RTP Profile for MPEG Video Payload
Fragmentation rules Video sequence header
if present, starts at the beginning of an RTP packet GOP sequence header
Either at beginning of RTP packet Or following video sequence header
Picture header Either at beginning of RTP packet Following GOP header
No header can span packets
Marker Bit Set to 1 if packet is end of picture
05. May 2004 18INF-3190: Multimedia
Protocols
RTP Profile for MPEG Video Payload
MPEG-1 Video specific payload header
MBZ Must be zero
TR Temporal reference The same number for all
packets of one frame For ordering inside an MPEG
GOP S
1 is sequence header is in this packet
B 1 if payload starts with new
slice E
1 if last byte of payload is end of slice
P 3 bits that indicate picture
type (I,P, or B) FBV, BFC, FFC, FFC
Indicate how a P or B frame is related to other I and P frames
MPEG Video Profile 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MBZ | TR |MBZ|S|B|E| P | | BFC | | FFC | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ FBV FFV
05. May 2004 19INF-3190: Multimedia
Protocols
RTP Quality Adaptation
Component interoperations for control of quality Evaluation of sender and receiver reports Modification of encoding schemes and parameters Adaptation of transmission rates Hook for possible retransmissions (outside RTP)
Application
RTCP RTP
DecodingEncoding
Application
UDP/IP UDP/IP
RTCPRTP
Decoding Encoding
05. May 2004 20INF-3190: Multimedia
Protocols
RTP Control Protocol (RTCP) Companion protocol to RTP (tight integration with RTP)
Monitoring of QoS of application performance
Feedback to members of a group about delivery quality, loss, etc.
Sources may adjust data rate Receivers can determine if QoS problems are local or network-wide
Loose session control Convey information about participants Convey information about session relationships
Automatic adjustment to overhead report frequency based on participant count
Typically, “RTP does ...” means “RTP with RTCP does ...”
05. May 2004 21INF-3190: Multimedia
Protocols
RTCP Packets
Several RTCP packets carried in one compound packet RTCP Packet Structure
SR Sender Report (statistics from active senders:bytes sent -> estimate rate)
RR Receiver Report (statistics from receivers) SDES Source Descriptions (sources as “chunks” with
several items like canonical names, email, location,...)
BYE explicit leave APP extensions, application specific
R SR / RR BYESDES APP
Compound (UDP) Packet
05. May 2004 22INF-3190: Multimedia
Protocols
RTCP Sender / Receiver Reports Sender report
Sender Information Timestamps Packet Count, Byte Count
List of statistics per source Receiver report
For each source Loss statistics Inter-arrival jitter Timestamp of last SR Delay between reception of last SR
and sending of RR Analysis of reports
Cumulative counts for short and long time measurements
NTP timestamp for encoding- and profile independent monitoring
HeaderSender Information
Reception Report
Profile Specific Extensions
Reception Report ...
Header
Reception Report
Profile Specific Extensions
Reception Report ...
05. May 2004 23INF-3190: Multimedia
Protocols
RTP Mixer Mixer
Reconstructs constant spacing generated by sender Translates audio encoding to a lower-bandwidth Mixes reconstructed audio streams into a single stream Resynchronizes incoming audio packets
New synchronization source value (SSRC) stored in packet Incoming SSRCs are copied into the contributing
synchronization source list (CSRC) Forwards the mixed packet stream Useful in conference bridges
05. May 2004 24INF-3190: Multimedia
Protocols
RTP Translator
Translation between protocols e.g., between IP and ST-2 Two types of translators are installed
Translation between encoding of data e.g. for reduction of bandwidth without adapting sources
No resynchronization in translators SSRC and CSRC remain unchanged
ATM UDP
ProtocolTranslator
MPEGSource
MPEGSink
H.263Sink
ProfileTranslator
05. May 2004 25INF-3190: Multimedia
Protocols
RTP Identifiers
S1 S3
S2 S4
M1 M2T1 R1
S1:10
S2:1
M1:33 (10,1)
M1:33 (10,1)
S4:13
S4:13
S3:19
M2:17 (19,13,33)
SSRC chosen by sender S1
Translators keep SSRCs and CSRCs
SSRC chosen by mixer M1
CSRCs from mixed sources S1 and S2
CSRCs contain previous SSRCs, but not previous CSRCs
05. May 2004 26INF-3190: Multimedia
Protocols
Protocol Development
Changes and extensions to RTP Scalability to very large multicast groups Congestion Control Algorithms to calculate RTCP packet rate Several profile and payload formats Efficient packetization of Audio / Video RTCP-based retransmission Loss / error recovery
05. May 2004 27INF-3190: Multimedia
Protocols
Non-QoSMultimedia Networking
Signalling Protocols:RTSP and SIP
05. May 2004 28INF-3190: Multimedia
Protocols
Signaling Protocols Applications differ
Media delivery controlled by sender or receiver Sender and receiver “meet” before media delivery
Signaling should reflect different needs Media-on-demand
Receiver controlled delivery of content Explicit session setup
Internet telephony and conferences: Bi-directional data flow, live sources (mostly) explicit session setup, mostly persons at both
ends Internet broadcast
Sender announces multicast stream No explicit session setup
05. May 2004 29INF-3190: Multimedia
Protocols
Real-Time Streaming Protocol (RTSP)
Internet media-on-demand Select and playback streaming media from server Similar to VCR, but
Potentially new functionality Integration with Web Security Varying quality
Need for control protocol Start, stop, pause, …
RTSP is also usable for Near video-on-demand (multicast) Live broadcasts (multicast, restricted control
functionality) ...
05. May 2004 30INF-3190: Multimedia
Protocols
RTSP Approach In line with established Internet protocols
Similar to HTTP 1.1 in style Uses URLs for addressing:
rtsp://video.server.com:8765/videos/themovie.mpg Range definitions Proxy usage Expiration dates for RTSP DESCRIBE responses Other referenced protocols from Internet (RTP, SDP)
Functional differences from HTTP Data transfer is separate from RTSP connection
typically via RTP Server maintains state – setup and teardown messages Server as well as clients can send requests
05. May 2004 31INF-3190: Multimedia
Protocols
RTSP Features Rough synchronization
Media description in DESCRIBE response Timing description in SETUP response Fine-grained through RTP sender reports
Aggregate and separate control of streams possible
Virtual presentations Server controls timing for aggregate sessions RTSP Server may control several data (RTP) servers
Load balancing through redirect at connect time Use REDIRECT at connect time
Caching Only RTSP caching so far Data stream caching is under discussion
05. May 2004 32INF-3190: Multimedia
Protocols
RTSP Methods
OPTIONSC S
determine capabilities of server/clientC S
DESCRIBE C S get description of media stream
ANNOUNCE C S announce new session description
SETUP C S create media session
RECORD C S start media recording
PLAY C S start media delivery
PAUSE C S pause media delivery
REDIRECT C S redirection to another server
TEARDOWN C S immediate teardown
SET_PARAMETER C S change server/client parameter
GET_PARAMETER
C S read server/client parameter
05. May 2004 33INF-3190: Multimedia
Protocols
media server
RTSP Integration
HTTPserver
RTSPserver
datasource
web browser
AVsubsystem
RTSPplug-in
HTTP GETpresentation description file
RTSP SETUPRTSP OK
RTSP PLAYRTSP OK
RTP AUDIO
RTP VIDEO
RTSP TEARDOWNRTSP OK
05. May 2004 34INF-3190: Multimedia
Protocols
Session Initiation Protocol (SIP) Lightweight generic signaling protocol Internet telephony and conferencing
Call: association between number of participants
Signaling association as signaling state at endpoints (no network resources)
Several “services” needed Name translation User location Feature negotiation Call control Changing features
05. May 2004 35INF-3190: Multimedia
Protocols
SIP Basics Call user Re-negotiate call parameters Forwarding (manual and automatic) Call center Supports personal mobility (change of terminal)
Through proxies or redirection Terminate / transfer calls ASCII (readable) protocol – SIP vs. H.323
Similarities (request/response, proxies ...) Differences (server state, server may initiate actions ...)
Control, location and media description (via SDP) Extensible towards
Usage for IP-IP, POTS-IP, inter-gateway interaction with firewalls, billing system, ...
Different modes Proxy mode Redirect mode
05. May 2004 36INF-3190: Multimedia
Protocols
SIP Operation – Proxy Mode
Proxy forwards requests Possibly in parallel to several hosts Cannot accept or reject call Useful to hide location of callee
User with“symbolic name”
calls another
1. Invite u@domain
7. Ok
Site A
8. ACK u@domain
11. Ok
4. Invite user@host
6. Ok
9. ACK user@host
10. Ok
5. “Ring”
2.
Where
?
3.
use
r@h
ost
Proxy ModeSite B
Location Server
05. May 2004 37INF-3190: Multimedia
Protocols
SIP Operation – Redirect Mode
User with“symbolic name”
calls another
2.
Where
?
3.
use
r@h
ost
Location Server
Redirect Mode1. Invite u@domain
4. Moved temporarilyLocation: user@domain2
5. ACK u@domain
6. Invite user@domain2 7. “Ring”
7. Ok
8. ACK user@domain2
Site A
Site B
05. May 2004 38INF-3190: Multimedia
Protocols
SIP – Methods Basic Methods (RFC 2543)
TABLE Additional Methods (partially
standardized) INFO: carry information between User Agents REFER: ask someone to send an INVITE to
another participant SUBSCRIBE: request to be notified of specific
event NOTIFY: notification of specific event