+ All Categories
Home > Documents > Multimedia SystemsCS-502 Fall 20071 Multimedia Systems CS-502 Operating Systems Fall 2007 (Slides...

Multimedia SystemsCS-502 Fall 20071 Multimedia Systems CS-502 Operating Systems Fall 2007 (Slides...

Date post: 16-Dec-2015
Category:
Upload: brenna-farrow
View: 217 times
Download: 0 times
Share this document with a friend
76
Multimedia System s CS-502 Fall 2007 1 Multimedia Systems CS-502 Operating Systems Fall 2007 (Slides include materials from Operating System Concepts, 7 th ed., by Silbershatz, Galvin, & Gagne and from Modern Operating Systems, 2 nd ed., by Tanenbaum)
Transcript

Multimedia SystemsCS-502 Fall 2007 1

Multimedia Systems

CS-502 Operating SystemsFall 2007

(Slides include materials from Operating System Concepts, 7th ed., by Silbershatz, Galvin, & Gagne and from Modern Operating Systems, 2nd ed., by Tanenbaum)

Multimedia SystemsCS-502 Fall 2007 2

Outline

• What is multi-media?• Requirements and challenges for audio and video in

computer systems– CD devices and formats

• Systems for multimedia• Compression and bandwidth• If time:–

– Processor and disk scheduling– Network streaming and management

Silbershatz, Chapter 20Tanenbaum, Chapter 7

Multimedia SystemsCS-502 Fall 2007 3

What do we mean by “multimedia”

• Audio and video within a computer system– CD’s & DVD’s– Computer hard drive– Interactive

• Live broadcast & web casts– Radio stations, Network TV, Webcams, Skype, …

• Video on demand– Pause, fast forward, reverse, etc.

• Interactive activities involving audio, video, images– Meetings and presentations with 2-way audio– Teleconferencing

• Handheld devices– iPod, MP-3 players; personal video; mobile phones, …

• …

Multimedia SystemsCS-502 Fall 2007 4

Multimedia Data and Delivery

• Stored in file system like ordinary data.• Must be accessed with specific timing

requirements.• E.g., video must be displayed at 24-30

frames per second.• System must guarantee timely delivery

• Continuous-media data – Data with specific rate requirements

Multimedia SystemsCS-502 Fall 2007 5

Multimedia Characteristics

• Multimedia files can be very large.

• Continuous media data may require very high data rates.

• Multimedia applications are usually sensitive to timing delays during playback.

• Note: human ear is more sensitive to jitter in audio than eye is to jitter in video!

Multimedia SystemsCS-502 Fall 2007 6

Requirements

• “Smooth” audio and video– Deterioration in quality jerky playback

• Multiple concurrent streams– Video & multimedia servers– TiVo, etc.

• Wide range of network bandwidths

• Audio/video while PC is doing something else

Multimedia SystemsCS-502 Fall 2007 7

System and OS Challenges

• Bandwidths and Compression

• Jitter

• Processor Scheduling

• Disk Scheduling

• Network Streaming

Multimedia SystemsCS-502 Fall 2007 8

Some System Architectures

• Simple: – Data paths for audio/video that are separate

from computational data paths

• Modern– Fast system bus, CPU, devices

• Video server– Disk farm and multiple streams

Multimedia SystemsCS-502 Fall 2007 9

audio stream

CD-ROM drive

System Organization (a decade ago)

• Separate data path for audio stream• Or headphone jack and volume control on CD drive itself

• Main system bus and CPU were too busy/slow to handle real-time audio

CPU

Memory

memory bus

Device Sound card

Multimedia SystemsCS-502 Fall 2007 10

video stream viaISA & bridge to

graphics card

audio stream viaISA bridge to

sound card

System Organization (typical Pentium PC today)

ISA bridge

IDE disk

Main Memory

CPULevel

2 cache

Bridge Moni-tor

Graphics card

USB

Key-board

Mouse

Ether-net

SCSI

ModemSoundcard

Printer

PCI bus

ISA bus

AGP Port

Multimedia SystemsCS-502 Fall 2007 11

Digression

CD and DVD devices and files

Multimedia SystemsCS-502 Fall 2007 12

Compact Discs (CDs)

• Not in Silbershatz– See Tanenbaum, pp. 306-310

• Audio CD– Molded polycarbonate; easy to manufacture– 120 mm diameter with 15 mm hole– One single spiral track

• Starts in center, spirals outward• 22,188 revolutions, approx 5.6 kilometers long• Random access

– Constant linear velocity under read head• Audio playback:– 120 cm/sec• Variable speed motor:– 200 – 530 rpm

• ISO standard IS 10149, aka the Red Book– Audio CD’s

Multimedia SystemsCS-502 Fall 2007 13

CDs for data storage

• Problem for adapting to data usage– No bad block recovery capability!

• ISO standard for data: Yellow Book– Three levels of error-correcting schemes: – Symbol, Frame, Sector

• ~7200 bytes to record 2048 byte payload per sector– Mode 2: less error correction in exchange for more data rate

• Audio and video data– Sectors linearly numbered from center to edge

• Read speed– 1x ~ 153,000 bytes/sec– 40x ~ 5.9 megabytes/sec

• ISO standard for multi-media: Green Book– Interleaved audio, video, data in same sector

Multimedia SystemsCS-502 Fall 2007 14

CD-ROM File SystemISO 9660 — High Sierra

• Write once contiguous file allocation• Variable length directories• Variable length directory entries

• Points to first sector of file• File size and metadata stored in directory entry• Variable length names

• Linux, Windows support iso9660 file system

• Several extensions to standard for additional features

• E.g., adding files, replacing directories

• See Tanenbaum, pp. 310-313

Multimedia SystemsCS-502 Fall 2007 15

DVD devices

• Similar in concept, different in details

• More data, higher bandwidth– Track spacing: 0.834μ vs. 1.6μ– Bit spacing: 0.4μ vs.

0.74μ– Capacity: 4.7 Gbytes vs.650 Mbytes

• Details not provided here– Tanenbaum, pp. 313-315

Multimedia SystemsCS-502 Fall 2007 16

Compression

An essential part of audio and video representation as files

Multimedia SystemsCS-502 Fall 2007 17

Why Compression? – CD-quality audio

• 22,050 Hz 44,100 samples/sec• 16 bits per sample

• Two channels 176,400 bytes/sec 1.4 mbits/sec

– Okay for a modern PC

– Not okay for 56 kb/sec modem (speed) or iPod (space)!

• MP-3 0.14 mbits/sec (10:1)– Same audio quality!

– Compression ratio varies with type of music

Multimedia SystemsCS-502 Fall 2007 18

Why Compression? – Video

• “Standard” TV frame = 640 480 pixels @ 25-30 frames/sec (fps) 9,216,000 pixels/sec = 27,648,000 bytes/sec

• Hollywood “standard” movie 133 minutes approx. 210 gigabytes (standard resolution)!

• HDTV = 1280 720 pixels @ 30 fps 82,944,000 bytes/sec (and rising!)

• DVD holds ~ 4.7 gigabytes average of ~ 620 kilobytes/sec!

• “Standard” movie of 133 minutes requires serious compression just to fit onto DVD

Multimedia SystemsCS-502 Fall 2007 19

Video Compression Requirements

• Compression ratio > 50:1• i.e., 210 gigabytes:4.7 gigabytes

• Visually indistinguishable from original• Even when paused

• Fast, cheap decoder• Slow encoder is okay

• VCR controls• Pause, fast forward, reverse

Multimedia SystemsCS-502 Fall 2007 20

Video Compression Standards

• MPEG (Motion Picture Experts Group)– Based on JPEG (Joint Photographic Experts Group)– Multi-layer

• Layer 1 = system and timing information• Layer 2 = video stream• Layer 3 = audio and text streams

• Three standards– MPEG-1 – 352240 frames; < 1.5 mb/sec ( < VHS quality)

• Layer 3 = MP3 Audio standard• Typical uses:– video clips on internet; video for handhelds

– MPEG-2 – standard TV & HDTV; 1.5-15 mb/sec• DVD encoding

– MPEG-4 – combined audio, video, interactive graphics• 2D & 3D animations

Multimedia SystemsCS-502 Fall 2007 21

JPEG compression (single frame)

1. Convert RGB into YIQ• Y = luminance (i.e., brightness) ~ black-white TV

• I, Q = chrominance (similar to saturation and hue)

Reason: Human eye is more sensitive to luminance than to color (rods vs. cones)

2. Down-sample I, Q channels• i.e., average over 22 pixels to reduce resolution

• lossy compression, but barely noticeable to eye

3. Partition each channel into 88 blocks• 4800 Y blocks, 1200 each I & Q blocks (for 640 480)

Multimedia SystemsCS-502 Fall 2007 22

JPEG (continued)

Multimedia SystemsCS-502 Fall 2007 23

JPEG (continued)

Multimedia SystemsCS-502 Fall 2007 24

JPEG (continued)

4. Calculate Discrete Cosine Transform (DCT) of each 88 block

• What is a Discrete Cosine Transform?

5. Divide 88 block of DCT values by 88 quantization table

• Effectively throwing away higher frequencies

6. Linearize 88 block, run-length encode, and apply a Huffman code to reduce to a small fraction of original size (in bytes)

Multimedia SystemsCS-502 Fall 2007 25

JPEG (concluded)

7. Store or transmit 88 quantization table followed by list of compressed blocks

• Achieves 20:1 compression with good visual characteristics

• Higher compression ratios possible with visible degradation

• JPEG algorithm executed backwards to recover image

• Visually indistinguishable from original @ 20:1

• JPEG algorithm is symmetric• Same speed forwards and backwards

Multimedia SystemsCS-502 Fall 2007 26

MPEG

• JPEG-like encoding of each frame• Takes advantage of temporal locality• I.e., each frame usually shares similarities

with previous frame encode and transmit only differences

• Sometimes an object moves relative to background find object in previous frame, calculate

difference, apply motion vector

Multimedia SystemsCS-502 Fall 2007 27

Temporal Locality (example)

Consecutive Video Frames

Multimedia SystemsCS-502 Fall 2007 28

MPEG organization

• Three types of frames– I-frame: Intracoded or Independent.

• Full JPEG-encoded frame• Occurs at intervals of a second or so• Also at start of every scene

– P-frame: Predictive frame• Difference from previous frame

– B-frame: Bidirectional frame• Like p-frame but difference from both previous and next frame

I B BB P B BB P B BB P B BB I B BB P B BB P

Multimedia SystemsCS-502 Fall 2007 29

MPEG Characteristics

• Non-uniform data rate!• Compression ratios of 50:1 – 80:1 are readily

obtainable• Asymmetric algorithm

– Fast decode (like JPEG)– Encode requires image search and analysis to get high

quality differences

• Cheap decoding chips available for – graphics cards– DVD players, etc.

Multimedia SystemsCS-502 Fall 2007 30

MPEG Problem – Fast Forward/Reverse

• Cannot simply skip frames– Next desired frame might be B or P derived

from a skipped frame

• Options:– Separate fast forward and fast reverse files

• MPEG encoding of every nth frame

• Often used in video-on-demand server

– Display only I and P frames• If B frame is needed, derive from nearest I or P

Multimedia SystemsCS-502 Fall 2007 31

“Movie” File Organization

• One MPEG-2 video stream• Possible fast forward, fast reverse sub-streams

• Multiple audio streams• Multiple languages

• Multiple text streams• Subtitles in multiple languages

• All interleaved• Possibly in multiple files!

Multimedia SystemsCS-502 Fall 2007 32

Multimedia SystemsCS-502 Fall 2007 33

Questions?

(skip to reading assignment)

Multimedia SystemsCS-502 Fall 2007 34

Operating System Challenge

• How to get the contents of a movie file from disk or DVD drive to video screen and speakers.– Fixed frame rate (25 or 30 fps)– Steady audio rate– Bounded jitter

• Such requirements are known as Quality-of-Service (QoS) guarantees.

• Classical problem in real-time scheduling– Obscure niche become mainstream!– See Silbershatz, §19.1–19.5

Multimedia SystemsCS-502 Fall 2007 35

QoS Guarantees

• Building a system to guarantee QoS effects the following:–– CPU processing– Scheduling and interrupt handling– File systems– Network protocols

Multimedia SystemsCS-502 Fall 2007 36

Requirement of Multimedia Operating Systems

• There are three levels of QoS– Best-effort service: the system makes a best

effort with no attempt to guarantee– Soft QoS: allows different traffic streams to be

prioritized, but no QoS guarantees are made– Hard QoS: system ensures QoS requirements

are always met• Prioritization• Admission control• Bounded latency on interrupt handling, processing,

etc.

Multimedia SystemsCS-502 Fall 2007 37

Parameters Defining QoS

• Throughput: the total amount of work completed during a specific time interval

• Delay: the elapsed time from when request is first submitted to desired resultJitter: delays that occur during playback of a stream.Reliability: how errors are handled during transmission and processing of continuous media

Multimedia SystemsCS-502 Fall 2007 38

Further QoS Issues

• QoS may be negotiated between the client and server.

• Operating systems may use an admission control algorithm– Admits a request for service only if the server

has sufficient resources to satisfy the request

Multimedia SystemsCS-502 Fall 2007 39

Two common methods

• Rate Monotonic Scheduling

• Earliest Deadline First

• Many variations

• Many analytic methods for proving QoS (Quality of Service)

Multimedia SystemsCS-502 Fall 2007 40

Processor Scheduling for Real-TimeRate Monotonic Scheduling (RMS)

• Assume m periodic processes– Process i requires ti msec of processing time

every pi msec.

– Equal processing every interval — like clockwork!

Multimedia SystemsCS-502 Fall 2007 41

Example• Periodic process i requires the CPU at specified intervals

(periods)• pi is the duration of the period• ti is the processing time• di is the deadline by when the process must be serviced

– Often same as end of period

Multimedia SystemsCS-502 Fall 2007 42

Processor Scheduling for Real-TimeRate Monotonic Scheduling (RMS)

• Assume m periodic processes– Process i requires ti msec of processing time every pi

msec.

– Equal processing every interval — like clockwork!

• Assume

• Assign priority of process i to be– Statically assigned

• Let priority of non-real-time processes be 0

11

m

i i

i

p

t

ip

1

Multimedia SystemsCS-502 Fall 2007 43

Rate Monotonic Scheduling (continued)

• Scheduler simply runs highest priority process that is ready– May pre-empt other real-time processes– Real-time processes become ready in time for

each frame or sound interval– Non-real-time processes run only when no real-

time process needs CPU

Multimedia SystemsCS-502 Fall 2007 44

Example

• p1 = 50 msec; t1 = 20 msec

• p2 = 100 msec; t2 = 35 msec

• Priority(p1) > Priority(p2)

• Total compute load is 75 msec per every 100 msec.

• Both tasks complete within every period• 25 msec per 100 msec to spare

Multimedia SystemsCS-502 Fall 2007 45

Example 2

• p1 = 50 msec; t1 = 25 msec

• p2 = 80 msec; t2 = 35 msec

• Priority(p1) > Priority(p2)

• Total compute load is ~ 94% of CPU.

• Cannot complete both tasks within some periods• Even though there is still CPU capacity to spare!

Multimedia SystemsCS-502 Fall 2007 46

Rate Monotonic Theorems (without proof)

• Theorem 1: using these priorities, scheduler can guarantee the needed Quality of Service (QoS), provided that

– Asymptotically approaches ln 2 as m ln 2 = 0.6931…

• Theorem 2: If a set of processes can be scheduled by any method of static priorities, then it can be scheduled by Rate Monotonic method.

)12(1

1

mm

i i

i mp

t

Multimedia SystemsCS-502 Fall 2007 47

Example 2 again

• Note that p1 pre-empts p2 in second interval, even

though p2 has the earlier deadline!

828.01229375.080

35

50

25 21

2

2

1

1

p

t

p

t

Multimedia SystemsCS-502 Fall 2007 48

More on Rate Monotonic Scheduling

• Rate Monotonic assumes periodic processes

• MPEG-2 playback is not a periodic process!

Multimedia SystemsCS-502 Fall 2007 49

Processor Scheduling for Real-TimeEarliest Deadline First (EDF)

• When each process i become ready, it announces deadline Di for its next task.

• Scheduler always assigns processor to process with earliest deadline.

• May pre-empt other real-time processes

Multimedia SystemsCS-502 Fall 2007 50

Earliest Deadline First Scheduling (continued)

• No assumption of periodicity• No assumption of uniform processing times

• Theorem: If any scheduling policy can satisfy QoS requirement for a sequence of real time tasks, then EDF can also satisfy it.– Proof: If i scheduled before i+1, but Di+1 < Di , then i

and i+1 can be interchanged without affecting QoS guarantee to either one.

Multimedia SystemsCS-502 Fall 2007 51

Earliest Deadline First Scheduling (continued)

• EDF is more complex scheduling algorithm• Priorities are dynamically calculated• Processes must know deadlines for tasks

• EDF can make higher use of processor than RMS

• Up to 100%

• There is a large body of knowledge and theorems about EDF analysis

Multimedia SystemsCS-502 Fall 2007 52

Example 2 (again)

• Priorities are assigned according to deadlines:

– the earlier the deadline, the higher the priority;

– the later the deadline, the lower the priority.

Multimedia SystemsCS-502 Fall 2007 53

Reading Assignments

• Silbershatz, §19.1–19.5– Real-time systems & Real-time scheduling

• Silbershat, Chapter 20– Multimedia systems

Multimedia SystemsCS-502 Fall 2007 54

Questions?

Multimedia SystemsCS-502 Fall 2007 55

Network Management

• General methods for delivering content from a server to a client across a network:–– Broadcasting: server delivers content to all

clients, whether they want it or not.– Multicasting: server delivers content to group

of clients who indicate they wish to receive the content.

– Unicasting: server delivers content to a single client.

Multimedia SystemsCS-502 Fall 2007 56

Network Management (continued)

• Broadcasting• Like cable TV

• Fixed portion of network bandwidth dedicated to set of broadcast streams; inflexible

• Multicasting• Typical webcast

• Multicast tree set up through internet to reach customers

• Customers can come and go

• Unicast• Dedicated connection between server and client

• Streaming version of familiar transport protocols

Multimedia SystemsCS-502 Fall 2007 57

Streaming –Delivery of Multimedia Data over Network

• Two types of streaming:–– Progressive download: client begins playback

of multimedia file during delivery. • File is ultimately stored on client computer

• (Hopefully) download speed > playback speed

– Real-time streaming: multimedia file is delivered to, but not stored on, client computer

• Played back at same speed as delivery

• Limited amount of buffering to remove jitter

Multimedia SystemsCS-502 Fall 2007 58

Two Types of Real-time Streaming

• Live streaming: to deliver a live event while it is occurring.

• Broadcast or multicast

• On-demand streaming: to deliver archived media streams

• Movies, lectures, old TV shows, etc.• Events not delivered when they occur.• Playback with pause, fast forward, reverse, etc• Unicast (usually)

Multimedia SystemsCS-502 Fall 2007 59

Network Streaming

• Traditional HTTP• Stateless

• Server responds to each request independently

• Real-Time Streaming Protocol (RTSP)• Client initiates a “push” request for stream

• Server provides media stream at frame rate

Multimedia SystemsCS-502 Fall 2007 60

Pull (HTTP) vs. Push (RTSP) server

Multimedia SystemsCS-502 Fall 2007 61

RTSP States

• SETUP: server allocates resources for client session.

• PLAY: server delivers stream to a client session.

• PAUSE: server suspends delivery of a stream.

• TEARDOWN: server releases resources and breaks down connection.

Multimedia SystemsCS-502 Fall 2007 62

RTSP state machine

Multimedia SystemsCS-502 Fall 2007 63

Bandwidth Negotiation

• Client application provides feedback to server to adjust bandwidth

• E.g., – Windows Media Player– RealPlayer– Quicktime

Multimedia SystemsCS-502 Fall 2007 64

Video Server

• Multiple CPUs

• Disk farm– 1000s of disks

• Multiple high-bandwidth network links– Cable TV– Video on demand– Internet

Multimedia SystemsCS-502 Fall 2007 65

Multimedia File & Disk Management

• Single movie or multimedia file on disk– Interleave audio, video, etc.

• So temporally equivalent blocks are near each other

– Attempt contiguous allocation• Avoid seeks within a frame

TextFrame

AudioFrame

Multimedia SystemsCS-502 Fall 2007 66

File organization – Frame vs. Block

• Frame organization• Small disk blocks (4-16 Kbytes)• Frame index entries point to starting block for each frame• Frames vary in size (MPEG)• Advantage: very little storage fragmentation• Disadvantage: large frame table in RAM

• Block organization• Large disk block (256 Kbytes or more)• Block index entries point to first I-frame of a sequence• Multiple frames per block• Advantage: much smaller block table in RAM• Disadvantage: large storage fragmentation on disk

Multimedia SystemsCS-502 Fall 2007 67

Frame vs. Block organization

smaller larger

Multimedia SystemsCS-502 Fall 2007 68

File Placement on Server

• Random

• Striped

• “Organ pipe” allocation– Most popular video in center of disk– Next most popular on either side of it, etc.– Least popular at edges of disk– Minimizes seek distance

Multimedia SystemsCS-502 Fall 2007 69

Resources on a file server

Multimedia SystemsCS-502 Fall 2007 70

Conclusion

• Multimedia computing is challenging• Possible with modern computers

• Compression is essential, especially for video

• Real-time computing techniques move into mainstream

• Processor and disk scheduling

• There is much more to this subject than fits into one class

Multimedia SystemsCS-502 Fall 2007 71

Reading Assignments

• Silbershatz, §19.1–19.5– Real-time systems & Real-time scheduling

• Silbershatz, Chapter 20– Multimedia systems

Multimedia SystemsCS-502 Fall 2007 72

Multimedia SystemsCS-502 Fall 2007 73

Digression on TransformsFourier’s Theorem

• Every continuous, periodic function can be reduced to the sum of a series of sine waves

• The Fourier transform is a representation of that function in terms of the frequencies of those sine waves

• Original function can be recovered from its Fourier transform

• Fourier transforms occur frequently in nature!

Multimedia SystemsCS-502 Fall 2007 74

Nyquist’s Theorem (1924)

• If a continuous function is sampled at a frequency 2f, then the function can be recovered from those samples provided that its maximum Fourier frequency is f.

Multimedia SystemsCS-502 Fall 2007 75

Discrete Cosine Transform

• A form of the Fourier Transform

• When applied to an 88 block of samples (i.e. pixel values) yields an 88 block of spatial frequencies

• Original 88 block of samples can be recovered from its DCT.– Assuming infinite arithmetic precision

Multimedia SystemsCS-502 Fall 2007 76

More on Nyquist• If arithmetic precision is not infinite, we get

quantization error during sampling• Recovered signal has quantization noise

• i.e., a lossy transform


Recommended