Multimedia Applications: Streamingce.sharif.edu/courses/89-90/2/ce873-1/resources... ·...

Multimedia Applications: Streaming

Hamid R. Rabiee

Mostafa Salehi, Fatemeh Dabiran, Hoda Ayatollahi

Spring 2011

Outline

² What is Streaming Technology?

² Issues in Video Streaming over the Internet

² Bandwidth

² Loss rate

² Delay

² Streaming types

² Streaming stored multimedia

² Streaming live multimedia

² Streaming through a web server

² Streaming through a streaming server

² Real Time Streaming Protocol (RTSP)

2 Digital Media Lab - Sharif University of Technology

What is Streaming Technology?² Streaming multimedia allows the user to begin viewing video clips without

completely downloading the entire file.

² After a brief initializing and buffering, the file begins to stream.

² Streaming process

² At Sender: Partition video into packets

² At Receiver: After X-second (e.g. 5s to 15s) delay -> begin playback while video is still being

downloaded

² At Receiver : Simultaneous delivery and playback (with short delay)

² Advantages:

² Low delay before viewing

² Minimum storage requirements

² Disadvantages:

² Any data that is lost or arrived late is useless


Issues in Video Streaming over the Internet

² Internet only offers best-effort service => Provide no guarantees on1. Bandwidth2. Loss rates3. Delay

1. Bandwidth² Available bandwidth is dynamic² Internet doesn’t provide bandwidth reservation

² If transmit faster than available bandwidth² Congestion occurs, packet loss, and severe drop in video quality

² If transmit slower than available bandwidth² Sub-optimal video quality

² Goal: Match video sending bit rate with available bandwidth = Rate Control² Video bit rate may be adapted by

² Varying the quantization² Varying the frame rate² Varying the spatial resolution² Adding/dropping layers (for scalable coding)

Digital Media Lab - Sharif University of Technology4

Issues in Video Streaming over the Internet : Overcoming Bandwidth Problem² Rate control schemes

² Source-based rate control

² Source is responsible for adapting the video transmission rate

² Applying in unicast video

² Probe-based approach: source probes the network for available

bandwidth by adjusting sending rate in a way that

Packet loss ratio p < threshold Pth

If p< Pth then increase transmission rate

If p> Pth then decrease transmission rate

² Model-based approach: It is based on the TCP throughput model to determine video stream’s sending

rate

² Applying in multicast video

² Sender uses single channel to transport video (single-channel multicast)

² Only probe-based approach can be employed

² All receivers share one channel => efficient but cannot provide various receivers demand with

different bandwidths


ρλ

××

=RTT

MTU22.1Time Trip RoundRTT , ratio lossPacket ρ

Unit Transmit MaximumMTU , TCP of Throughputλ==

==

Fig.1 – An architecture for the source-based rate control system

Match the rate of pre-compressed video bitstream to the target rate

Issues in Video Streaming over the Internet : Overcoming Bandwidth Problem (cont.)² Receiver-based rate control

² Receiver is responsible for adapting the video

receiving rate by adding/dropping channels

² Useful in multicasting scalable video; each video

layer correspond to one channel in multicast tree

² Applying approaches

² Probe-based

² No congestion detected => receiver probes for available bandwidth => joining a layer => receiving rate

increased

² Congestion detected => receiver drops a layer => receiving rate reduced

² Model-based

² The same as in the sender-based rate control

² Hybrid rate control

² Receivers regulate the receiving rate of video streams by adding/dropping channels while sender adjusts the

transmission rate of each channel based on the receivers’ feedback

² Examples: Destination set grouping, Layered multicast scheme


Source

Client with high-rate connection

Client with medium-rate connection

Client with low-rate connection

Client with low-rate connection

Base layer

Enh layer 1

Enh layer 2

Fig.2 – Example of Receiver-Driven Layered Multicast

Issues in Video Streaming over the Internet : Overcoming Loss Rate Problem2. Packet Loss² Can damage pictures in a video sequence² Goal: Overcome packet loss via error control²Forward Error Correction (FEC)

² A video stream chopped into segments, each segment is packetized into k packet

² A block code applied to k packets => n-packet block generated² By receiving any k packets in n-packet block, the segment can be

recovered² Advantages:

² Low delay (as compared to retransmits)² Doesn’t require feedback channel²Works well (if appropriately matched to channel)

² Disadvantages:²Overhead² Channel loss characteristics are often unknown and time-varying² FEC may be poorly matched to channel. Therefore, it is often

ineffective (too little FEC) or inefficient (too much FEC)


Issues in Video Streaming over the Internet : Overcoming Loss Rate Problem

3

1 2

41 3

1 2 3 4

1 1 2 2 3 4

12 LOSS

3 4

Original stream

Redundancy

Received stream

Reconstructed stream

Internet


² FEC Mechanism

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1 5 9 13 2 6 10 14 3 7 11 15 4 8 12 16

1 5 9 13 2 6 10 14 LOSS 4 8 12 16

1 2 4 5 6 8 9 10 12 13 14 16

Original Stream

Interleaved StreamReceived Stream

Reconstructed Stream


Issues in Video Streaming over the Internet : Overcoming Loss Rate Problem (cont.)² Interleaving Mechanism

Issues in Video Streaming over the Internet : Overcoming Loss Rate Problem (cont.)² Retransmission

² When receiver detects the loss of packet N:if ( Tc + RTT + Ds < Td (N)) then “send the request for packet N to sender”

{ Tc = Current time, RTT= Round trip time, Ds= Slack term, Td(N)= time of scheduling packet N for display }

² Two approaches in video streaming with time-sensitive data² Delay-constrained retransmission

² Only retransmit packets that can arrive in time² Priority-based retransmission

² Retransmit important packets before unimportant packets² Advantages:

² Only resends lost packets, efficiently uses bandwidth² Easily adapts to changing channel conditions

² Disadvantages:² Latency (round-trip-time (RTT))² Requires a back-channel (not applicable in broadcast, multicast, or point-to-

point w/o back-channel)² For some applications, usefulness decreases with increasing RTT


Issues in Video Streaming over the Internet : Overcoming Loss Rate Problem (cont.)

² Error concealment

² Performed by receiver when packet loss has already occurred

² Approaches to apply error concealment² Spatial interpolation: Missing pixels are reconstructed using neighboring

spatial information² Temporal interpolation: loss data is reconstructed using data in previous

frames

² Error-resilient video coding

² Using Multiple Description Coding (MDC): a raw video sequence is compressed

into multiple streams

² Robust to packet loss: if receiver gets one description, it can reconstruct video with

acceptable quality

² Enhanced video quality: if receiver gets multiple description, a better reconstruction can

be made by combining them


Issues in Video Streaming over the Internet : Overcoming Delay Problem3. Delay (Delay Jitter)² Delay jitter:²End-to-end delay may fluctuate from packet to packet

²Streaming video needs bounded end-to-end delay so that packets can arrivein time for decoding and playback

²Jitter: Variation in the end-to-end delay² Receiver should decode and display frames at the same rate²Each frame has its own specific playout time²Playout time: Deadline by which it must be

received/decoded/displayed² If video packet arrives late (after its plaback deadline) => it is

useless , can be considered as lost² If subsequent frames depend on the late frame, then effects can

propagate² Goal: Overcome delay jitter => Playback buffer


Issues in Video Streaming over the Internet : Overcoming Delay Problem (cont.)


² Approach: Add playback buffer at decoder side to compensate for jitter

² Corresponds to adding an offset to the playback time of each packet² If (packet delay < offset) then OK²Buffer packet until its playback time

² If (packet delay > offset) then problem² Playback buffer typically has duration of 5-15 secs² Compensates for delay jitter and enables retransmission of lost packets

constant bit rate video

transmission

time

variablenetwork

delay (jitter)

client videoreception

constant bit rate video

playout at client

client playoutdelay

buffe

red

vide

o



² Client-side buffering:


² Jitter: Some Definitions

² Input: t0, …, tn

² Delay Jitter:

² Delay jitter J

² For every k: |t0 + kX - tk | ≤ J

² X = (tn - t0) / n

² Rate Jitter

² Rate Jitter A

² Ik = tk - tk-1

² For every k and j: |Ij - Ik| ≤ A

² Jitter and buffering

² delay versus jitter


Issues in Video Streaming over the Internet : Overcoming Delay Problem (cont.)² Packet delivery, time-varying delay (jitter), and

playout delay:


Issues in Video Streaming over the Internet : Overcoming Delay Problem (cont.)² Effect of Different Playout Delays

² Playout delays: TD1 < TD2 < TD3


Streaming Stored Multimedia

Streaming stored media:

² Audio/video file is stored in a

server

² Users request audio/video file

on demand.

² Audio/video is rendered

within, say, 10 s after request.

² Interactivity (pause, re-

positioning, etc.) is allowed.

Media player:

² removes jitter

² decompresses

² error correction

² graphical user interface

with controls for

interactivity

² Plug-ins may be used to

embed the media player into

the browser window.


Streaming Stored Multimedia (cont.)


1. videorecorded

2. videosent

3. video received,played out at client

streaming: at this time, client playing out early part of video, while server still sending laterpart of video

networkdelay

time

Streaming Live Multimedia

² Broadcasting the media, live over the Internet. The process involves ² a camera for the media, ² an encoder to digitize the content, ² a media publisher where the streams are made available to potential end-users ² a content delivery network to distribute and deliver the content.

² playback buffer is required (as with the streaming stored multimedia)² playback can lag tens of seconds after transmission² still have timing constraintInteractivity Features² fast forward impossible² rewind, pause possibleExamples:² Internet radio talk show² Live sporting event


Interactive, Real-Time Multimedia

² End-end delay requirements:² audio: < 150 msec good, < 400 msec OK

² includes application-level (packetization) and network delays² higher delays noticeable, impair interactivity

² Session initialization² how does callee advertise its IP address, port number, encoding algorithms?

² Applications: IP telephony, video conference, distributed interactive worlds


Interactive, Real-Time Multimedia (cont.)

² Applications:

² PC-2-PC phone

²instant messaging services are providing this

² PC-2-phone

²Dialpad

²Net2phone

² Videoconference with Webcams



Internet Phone, an example of interactive multimedia

² speaker’s audio: alternating talk spurts, silent periods.

² 64 kbps during talk spurt

² pkts generated only during talk spurts

² 20 msec chunks at 8 Kbytes/sec: 160 bytes data

² application-layer header added to each chunk.

² Chunk+header encapsulated into UDP segment.

² application sends UDP segment into socket every 20 msec during

talkspurt.



Internet Phone (cont.)

² Network loss: IP datagram lost due to network congestion (router buffer

overflow)

² Delay: IP datagram arrives too late for playout at receiver => considered

as lost

² delays: processing, queueing in network; end-system (sender, receiver) delays

² Typical maximum tolerable delay: 400 ms

² loss tolerance: depending on voice encoding, losses concealed, packet loss

rates between 1% and 10% can be tolerated.



packets

time

packetsgenerated

packetsreceived

loss

r

p p'

playout schedulep' - r

playout schedulep - r

Internet Phone: Fixed Playback DelayØ Sender generates packets every 20 msec during talk spurt.Ø First packet received at time rØ First playback schedule: begins at pØ Second playback schedule: begins at p’



packetith receivingafter delay network average of estimatedacketpith for delay network tr

receiverat played is ipacket timethepreceiverby received is ipacket timether

packetith theof timestampt

i

ii

i

i

i

==−

===

)()1( 1 iiii trudud −+−= −

Adaptive Playout Delay

² Goal: minimize playout delay, keeping late loss rate low

² Approach: adaptive playout delay adjustment:

² Estimate network delay, adjust playout delay at beginning of each talk spurt.

² Silent periods compressed and elongated.

² Chunks still played out every 20 msec during talk spurt.

² Dynamic estimate of average delay at receiver (u is a fixed constant ):



||)1( 1 iiiii dtruvuv −−+−= −

iiii Kvdtp ++=

Adaptive Playout Delay (cont.)

ØAlso useful to estimate the average deviation of the delay, vi :

ØThe estimates di and vi are calculated for every received packet, although they are only used at the beginning of a talk spurt.

ØFor first packet in talk spurt, playout time is:

where K is a positive constant.

ØRemaining packets in talkspurt are played out periodically


Streaming from a Web server

² Audio and video files stored in Web servers

naïve approach

² browser requests file with HTTP request

message

² Web server sends file in HTTP response

message

² content-type header line indicates an

audio/video encoding

² browser launches media player, and passes file

to media player

² media player renders file

Major drawback: media playerinteracts with server throughintermediary of a Web browser


Streaming from a Web server (cont.)

Alternative: set up connection between

server and player

² Web browser requests and receives a

meta file

(a file describing the object) instead

of receiving the file itself;

² Content-type header indicates

specific audio/video application

² Browser launches media player and

passes it the meta file

² Player sets up a TCP connection with

server and sends HTTP request.

Some concerns:

² Media player communicates over

HTTP, which is not designed with

pause, ff, rwnd commands

² May want to stream over UDP

² These concerns may not be real

problems – more later


Streaming from a streaming server

² This architecture allows for

non-HTTP protocol between

server and media player

² Can also use UDP instead of

TCP.


Streaming Multimedia: UDP or TCP?

UDP ² server sends at rate appropriate for client (oblivious to network

congestion !)² often send rate = encoding rate = constant rate² then, fill rate = constant rate - packet loss

² short playback delay (2-5 seconds) to compensate for network delay jitter

² error recover: time permitting

TCP² send at maximum possible rate under TCP² fill rate fluctuates due to TCP congestion control² larger playback delay: smooth TCP delivery rate² HTTP/TCP passes more easily through firewalls


Real Time Streaming Protocol (RTSP) ² A signaling and control protocol for multimedia streaming in Internet² To control the data delivery in a multimedia streaming session by conveying

VCR-style commands (like play, mute) between communicating partners² It is typically used in conjunction with RTP which conveys the actual

multimedia data.² It is a request-response protocol similar to HTTP, but stateless² Server and player use RTSP to send control info to each other

² For user to control display: rewind, fast forward, pause, resume, repositioning, etc…

What it doesn’t do:² Does not define how audio/video is encapsulated for streaming over

network² Does not restrict how streamed media is transported; it can be

transported over UDP or TCP² Does not specify how the media player buffers audio/video


RTSP Request² Has the form:

Request-Method SP Request-URL SP RTSP-Version <CR><LF>(generic-header | request-header | entity-header <CR><LF>)<CR><LF>[message body]

² Request-Method is:² DESCRIBE - retrieves the description of a media object from a

server;² SETUP - prepares the streaming session;² PLAY - starts the delivery of multimedia data;² PAUSE - streaming is paused, session is still active, but no packet is

sent;² TEARDOWN - session is terminated and resources are freed.


RTSP Request (cont.)² Request-header can have the following fields (selection):

² Accept : MIME types of resources accepted by client

² Accept-Encoding : encoding accepted by client

² Accept-Language : language accepted by client

² Authorization : user-agent wishes to authenticate itself with a

server

² From:

² Referer : the URL of document refering this URL

² User-Agent : client software


RTSP Response² Has the form:

Http-Version SP Status-Code SP Reason-Phrase<CR><LF>(generic-header | response-header | entity-header <CR><LF>)<CR><LF>[message body]

² Response-header has the following fields (selection):² Location : redirect the client to a location other than Request-URL

for completion of the request² Retry-After : indicate to client how long the service is expected to be

unavailable² Server : information about software used by the server to handle the

request


RTSP: out of band control

FTP uses an “out-of-band” control

channel:

² A file is transferred over one channel.

² Control information (directory changes, file

deletion, file renaming, etc.) is sent over a

separate TCP connection.

² The “out-of-band” and “in-band” channels

use different port numbers.

RTSP messages are also sent out-of-band:

² The RTSP control messages use different

port numbers than the media stream, and

are therefore sent out-of-band.

² The media stream, whose packet structure

is not defined by RTSP, is considered “in-

band”.

² If the RTSP messages were to use the same port

numbers as the media stream, then RTSP

messages would be said to be “interleaved” with

the media stream.


RTSP initiates and controls delivery² Client obtains a description of the multimedia

presentation, which can consist of several media streams.

² The browser invokes media player (helper application)

based on the content type of the presentation description.

² Presentation description includes references to media

streams, using the URL method rtsp://

² Player sends RTSP SETUP request; server sends RTSP

SETUP response.

² Player sends RTSP PLAY request; server sends RTSP

PLAY response.

² Media server pumps media stream.

² Player sends RTSP PAUSE request; server sends RTSP

PAUSE response.

² Player sends RTSP TEARDOWN request; server sends

RTSP TEARDOWN response.

HTTP GET

SETUP

PLAY

media stream

PAUSE

TEARDOWN

mediaplayer

Webserver

mediaserver

Webbrowser

client server

presentation desc.

Meta file example<title>Twister</title>

<session>

<group language=en lipsync>

<switch>

<track type=audio

e="PCMU/8000/1"

src = "rtsp://audio.example.com/twister/audio.en/lofi">

<track type=audio

e="DVI4/16000/2" pt="90 DVI4/8000/1" src="rtsp://audio.example.com/twister/audio.en/hifi">

</switch>

<track type="video/jpeg"

src="rtsp://video.example.com/twister/video">

</group>

</session>


RTSP session

² Each RTSP has a session identifier,

which is chosen by the server.

² The client initiates the session with

the SETUP request, and the server

responds to the request with an

identifier.

² The client repeats the session

identifier for each request, until the

client closes the session with the

TEARDOWN request.

² RTSP port number is 554.

² RTSP can be sent over UDP or

TCP. Each RTSP message can be

sent over a separate TCP

connection.


RTSP: exchange example

C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0 Transport: rtp/udp; compression; port=3056; mode=PLAY

S: RTSP/1.0 200 1 OK Session 4231

C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=0-

C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=37

C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231

S: 200 3 OK

npt = Normal play time


RTSP: streaming caching² Caching of RTSP response messages

makes little sense.

² But desirable to cache media streams

closer to client.

² Much of HTTP/1.1 cache control has

been adopted by RTSP.

² Cache control headers can be

put in RTSP SETUP requests

and responses:

² If-modified-since: , Expires: ,

Via: , Cache-Control:

² Proxy cache may hold only segments of

a given media stream.

² Proxy cache may start serving a

client from its local cache, and

then have to connect to origin

server and fill missing material,

hopefully without introducing

gaps at client.

² When original server is sending a

stream to client, and stream passes

through a proxy, proxy can use TCP to

obtain the stream; but proxy still sends

RTSP control messages to origin server.


Next Session

Multimedia Protocols


Date post:	13-Jul-2020
Category:	Documents
Upload:	others
View:	5 times
Download:	0 times

Multimedia Applications: Streamingce.sharif.edu/courses/89-90/2/ce873-1/resources... ·...

Documents