1

Chapter 1: Overview

2

What is multimedia?

Multimedia = too many cables? Multimedia combines audio and visual materials to

provide computerized interaction of text, sound, graphics, images, animation & video to enhance communication and to enrich its presentation.

Multimedia systems handle at least one type of “continuous media” as well as “static media”.

3

Multimedia applications

Videodisc applications A DVD can hold 2-8 hrs of high-quality video.

Electronic games Web browsers Multimedia presentation systems

An “engine” that displays, synchronizes, provides interaction with, and generally manipulates multimedia material (e.g., Macromedia Flash).

4

Multimedia applications

Multimedia mail systems. Teleconferencing

A computer equipped with microphone, speakers, and a video camera, and placed on a multimedia network, can establish audio and video connections between other, similarly equipped, machines.

Multi-user tools, such as group editors. A group editor allows conference participants to share documents, and to edit the documents simultaneously.

5

Multimedia applications

Multimedia services Interactive TV Interactive shopping Education Medical services (telemedicine) Video-on-demand

12

Analog vs. Digital

Two ways to process information: analog and digital

Examples: Vinyl LP vs. CD conventional radio vs. web-radio slide-rule vs. digital calculator

13

Analog recording

Sound is caused by a variation of air pressure – a sound wave

14

Analog recording

An audio tape records an analog signal that represents the sound wave.

An analog signal is continuous – there are an infinite number of points (values) and each value has infinite precision.

15

Digital recording

To record an analog signal by a computer (which has a finite amount of storage), we need to perform analog-to-digital conversion (ADC): Sampling Quantization Coding

16

ADC

Consider an analog signal

time

17

Sampling

convert continuous signal into discrete values by taking samples

18

Sampling

The original signal can be approximated by interpolation using the sampled values.

19

Sampling

More samples more accurate approximation

20

Quantization

A computer cannot record the values with infinite precision. A value has to be quantized.

a quantizationLevel/interval

21

Quantization

A computer cannot record the values with infinite precision. A value has to be quantized.

a coarserquantization

22

Quantization

Each sample value is replaced by the nominal value of its quantization level.

a sample value

a nominalvalue

23

Quantization

Each sample value is replaced by the nominal value of its quantization level.

24

Quantization

The difference between a sample value and its quantization value is called the quantization error.

quantization error

25

Quantization

More quantization levels gives a more accurate representation.

26

Quantization

More quantization levels gives a more accurate representation.

27

Quantization

More quantization levels gives a more accurate representation.

28

Quantization

Note that quantization does not necessarily have to be uniform or linear.

widerquantizationlevel

narrowerquantizationlevel

29

Coding

assign a codeword to each quantization level.

00001000

1101

1001

0101010000110010

0001

101010111100

30

Coding

An analog signal could then be represented digitally by a string of 0’s and 1’s 0010 0011 0011 0000 1010 1011 … …

31

Why digital?

ADC introduces error The waveform constructed by interpolating samples is not a verbatim

copy of the original. Quantization introduces quantization error.

so … Why digital representation? What should be the sampling frequency? How many quantization levels (or bits/sample) shall we use?

Example: for CD-DA (music CD): sampling frequency: 44,100 samples per second 16 bits per sample (i.e., 65,536 quantization levels)

32

Why digital?

easy integration & sharing of resources (storage, transmission network).

can be processed by a computer, e.g., encryption, watermarking, compression, digital effects, etc.

more “reliable” storage and transmission (through error-correction and replication).

can be used and copied many times without losing quality.

33

Why digital?

channel channel analog signal transmission

channel reconstruct digital signal transmission

34

Media types

Non-temporal (Discrete) – do not have a time dimension, and their contents & meanings do not depend on the presentation time. Text Image Graphics

35

Media types

Temporal (Continuous, Isochronous) – have a time dimension. They convey meanings only if “displayed” at a specific rate. Video Digital Audio Music

36

Text

not visually exciting conveys essential and precise information text representation: e.g., ASCII, BIG5, GB storage “friendly” sometimes, certain information is too

abstract to be captured by words

37

Digital images

To digitize a photo, we again perform ADC

38

Digital images

sampling

39

Digital images

pick a sample color from each “box” to record

Each sample is called a pixel picture element

40

Digital images

finer sampling gives a clearer image

41

Digital images

finer sampling yet

42

Digital images

The resolution of a digital camera refers to the number of samples that the camera takes.

Some typical numbers:

No. of samples Resolution Print size

0.3 M 640 480 3 4 inches

1 M 1152 864 5 7 inches

1.2 M 1280 960 5 7 inches

2.1 M 1600 1200 8 10 inches

3.3 M 2048 1536 11 14 inches

You needat least 150ppi(pixel per inch)for printing

For better quality,make it 300 ppi

43

Digital cameras

A Charge-coupled-device (CCD) camera has a 2-dimensional array of photosites that convert the amount of light intensities into equivalent electrical charges. Note: there are also less-expensive digital cameras that

use CMOS instead of CCD. CMOS sensors are usually: More susceptible to noise Less light-sensitive Consume less power

Both technologies have advanced in recent years that they are comparable against each other now.

44

Capturing color

There are a number of ways that a CCD camera captures color: Split a light beam into its red, green and blue components and use 3

separate CCD grids.

45

Capturing color

There are a number of ways that a CCD camera captures color: A more economical way is to use 1 CCD panel

and cover each photosite with a different filter.

46

Capturing color

R G BB R GG B R

The amount of charge at each photosite tells the intensity ofa color component of light detected at that point.

47

Capturing color

R G BB R GG B R

red

48

Capturing color

R G BB R GG B R

green

49

Capturing color

R G BB R GG B R

blue

50

Capturing color

Bayer filter

51

Digital images

two-dimensional arrays of pixels (picture elements) of varying color and intensity.

color model: how to specify the color of a pixel (coding)? RGB: colors can be represented by numeric triplets specifying

red (R), green (G), and blue (B) intensities. Y’CRCB (for digital images and video) CMY(K) (for printing)

Cyan (no red) Magenta (no green) Yellow (no blue) K — black ink

Additivemixing

Colordifferences

Subtractivemixing

52

Digital images

Compression A page-sized 24-bit color image with 300 pixels per inch

(PPI) takes up about 20Mbytes. Lossless and lossy compression. Many different standards: JPEG, GIF (Graphic

Interchange Format), ...Example: run-length encoding.

Image transformation and processing. E.g., morphing (one image transformed into another).

53

Graphics data are represented by a geometric model + a set of graphics operations.

Geometric model A collection of 2D/3D geometric primitives

(lines, circles, polygons, curves). Transformations: rotation, translation, scaling.

Graphics

54

Graphics

55

3D models

taken fromPlanetArchitecture

Wireframe

56

Graphics operations are applied to make the scene more realistic: Coloring Shading

How a surface reflect light Lighting

Ambient light (from all directions) Point light (inverse square law) Spot light (a cone-shaped volume)

Viewing Where the camera is

Graphics

57

Graphics

Texture mapping applying an image onto a surface

Renderingconverts a model + shading, lighting + viewing ... into

an image.

Animation Eye-catching Good for demonstration

58

Graphics

Taken fromSIGGRAPH’ 97

59

Graphics

60

Graphics

61

Graphics

taken fromPlanetArchitecture

62

Analog video

A sequence of images called frames, “persistence of vision”.

Originally, motion pictures are shown at an (insufficient) frame rate of 16fps.

It was found that for smooth motion, 24fps is needed.

Attributes: frame rate, resolution, aspect ratio, interlacing, refresh rate.

63

Analog video

Formats. Examples:

NTSC (National Television Systems Committee) PAL (Phase Alternation Line).

format frame rate scan lines aspect ratioNTSC 30 525 4:3PAL 25 625 4:3HDTV(US) 30 1125 16:9HDTV(EURO) 25 1250 16:9

MUSE(Japan) 30 1125 16:9

picturewidth topictureheight

64

Analog video

Theoretically, most color can be produced by mixing 3 primary colors (red, green, blue). An analog video camera produces 3 distinct continuous signals, one for each color component.

black

white

horizontalblanking

scan linesignal

white

black…

vertical blanking

65

Analog video

Theoretically, most color can be produced by mixing 3 primary colors (red, green, blue). An analog video camera produces 3 distinct continuous signals, one for each color component.

black

white

horizontalblanking

scan linesignal

white

black…

vertical blanking

66

Analog video

Theoretically, most color can be produced by mixing 3 primary colors (red, green, blue). An analog video camera produces 3 distinct continuous signals, one for each color component.

black

white

horizontalblanking

scan linesignal

white

black…

vertical blanking

67

Analog video

Theoretically, most color can be produced by mixing 3 primary colors (red, green, blue). An analog video camera produces 3 distinct continuous signals, one for each color component.

black

white

horizontalblanking

scan linesignal

white

black…

vertical blanking

68

Analog video

A video signal is applied to a TV (a CRT) to control the power of an electron beam that strikes the phosphors on the inside of the CRT surface.

With the 625/50 system (PAL), for example, a frame is displayed every 1/25 seconds.

Unfortunately, the phosphors do not stay lit that long flickering

To prevent flicker, the picture has to be refreshed at least 50 times per second.

Interlacing: a frame is divided into 2 fields: odd-lines and even-lines, a field is displayed every 1/50 seconds.

69

Analog video

time0 1/25

frame 1 frame 2

1/50

progressive

interlaced

70

Analog video

Luminance/chrominance principle: the three primary colors can be converted into 2 parts: Luminance: information on the lightness of the image. Chrominance: information on the color of the image.

Because the human eye is not very sensitive to color information, the bandwidth of the color component is reduced before transmission.

71

Analog video

Video storage video tape e.g., VHS tape

about 240 scan lines resolution problem: loss of quality when copying and repeated

playback due to stretches and magnetic material wearing off.

72

Digital video

A video can also be represented by a sequence of digital images.

broadcast quality video (uncompressed): 1 sec > 20 MB

For lesser quality, and a good compression technique, it is possible to achieve:

1 sec = 1.2 Mb transfer rate of (single-speed) CD-ROM VCD

Data rate of a DVD movie is about 2GB/hr. Compression: lossless and lossy.

B – byteb – bit

73

Digital video

For lossy compression, we can achieve a 50:1 or higher compression ratio.

MPEG: The Moving Pictures Expert Group MPEG-1: 1.5Mbps VHS quality video MPEG-2: 4-10 Mbps (digital TV) MPEG-4: a system which allows a scene structure to be

composed of multiple different objects (video, audio, natural, synthetic)

MPEG-7: multimedia information retrieval

74

Digital video formats

CCIR 601 (4:2:2 chromatic subsampling) for video exchange 525/60 system: 720 480 (take 720 samples from each active scan

line) 625/50 system: 720 576

4:2:0 for video broadcast

75

Digital video formats

SIF (Standard Input Format) for storage 525/60 system: 352 240 625/50 system: 352 288 4:1:1 chromatic subsampling

CIF (Common Interchange Format) for video conferencing 352 288 at 30 fps using 4:1:1 chromatic subsampling

76

Digital audio

Digital audio representation produced by sampling a continuous signal

generated by a sound source ADC (again)

77

Digital audio

Sampling frequency Nyquist’s Theorem

Sampling rate 2 highest signal frequency Human ear is sensitive to frequencies of up to about

20kHz (c.f., rat: 1k – 10k Hz; cat: 100 – 60k Hz) Sampling frequency 40kHz

audio-CD 44.1kHz, 16 bits per sample ( 96dB max. SNR) DVD audio 196kHz (max), 24 bits per sample (max) (Q: what is the throughput requirement?)

For stereo, 2 channels.

78

Digital audio

Storage An hour of high quality stereo digital audio

requires > 500MBytes of storage. A CD-ROM can store about 650MBytes of data. A CD-DA can store about 74 minutes of audio

data.

79

Digital audio

Why 16 bits per sample? Let b = number of bits per sample, q = quantization step,

Q = number of quantization levels. We have Q = 2b. Max. signal amplitude = (q2b)/2; Max. quantization noise = q/2. SNR = 20log10(q2b/q) dB 6b dB. Threshold-of-pain / audibility-threshold = 100 to 120 dB. Quantization noise is inaudible SNR is at least 100. 6b = 100 b = 16.7.

80

Digital audio

q

2b levels

(q2b)/2

81

Music

MIDI -- Musical Instrument Digital Interface Digital musical instruments send MIDI messages to a

sequencer. The sequencer composes the music according to the

messages received. The sequencer/synthesizer has a “palette” of sounds for

each type of instrument

MIDI sequencer

82

Challenges

Multimedia stresses all components of a computer system (data volume & time constraints)

CPU processing power fast speed for data capturing, CODEC, data enhancement large amounts of data being processed in real-time

Storage and Memory high capacity, fast access time, high transfer rates

System architecture high bus bandwidth, efficient I/O

83

Challenges

Software tools for retrieval and data management of continuous

media data

Operating systems support for new data types, real-time scheduling,

multimedia file systems, time-critical synchronization

Networks high bandwidth, low latency, low jitter

84

Research areas

1. fast processors

2. high-speed networks

3. large capacity storage devices

4. video & audio compression algorithms

5. graphics systems

6. human-computer interfaces

7. real-time operating systems

8. information storage and retrieval

9. hypertext & hypermedia

10. languages for scripting

11. parallel processing methods

85

Compression

MM systems require data compression for 3 reasons: the large storage requirements of MM data. relatively slow storage devices that cannot play

MM data in real time network bandwidth that does not allow real-time

video data transmission

86

Some compression standardsJPEG Digital

compression andcoding ofcontinuous-tonestill images

JointPhotographicExperts Group

15:1 (full color still-frame applications)

H.261 Videocoder/decoder foraudio-visualservices at p*64Kbps

SpecialistGroup onCoding forVisualTelephony

100:1 to 2000:1 (video-basedtelecommunications)

MPEG Coding of movingpictures andassociated audio

MovingPicturesExperts Group

200:1 Motion-intensiveapplications

87

Multimedia networking

Many multimedia applications, such as video mail, video conferencing, and video-on-demand, require the support of a high performance network system.

In these applications, the multimedia objects are stored at a server and played back at the clients’ sites.

Remote retrieval of multimedia objects has stringent time constraints.

88

Multimedia networking

Delay: the amount of time it takes to transmit a data unit (e.g., a video frame) from a sender to a receiver.

Jitter: delay variation.

source destination

delay

jitter

89

Multimedia networking

Characteristics Data Transfer Multimedia transfer

Data rate Low High

Traffic pattern Bursty Stream-oriented

Reliabilityrequirements

No loss Some loss

Latencyrequirements

None Low, e.g., 20 ms

Mode ofcommunication

Point-to-point Multipoint

Temporalrelationship

None Synchronizedtransmission

90

Multimedia information retrieval

To retrieve a text document from the Web, we use keyword search via “Alta Vista”, for example.

To retrieve a record from a relational database, such as Oracle, we use an SQL statement.

How shall we formulate a query to retrieve pictures?

What about audio? How do we describe a sound? How do we describe a song?

91

Multimedia software tools

Music sequencing Cakewalk

supports general MIDIprovides several editing views (staff, piano roll, event

list) and Virtual Piano

92

Multimedia software tools

Image and video editing Adobe Photoshop

allows layering of images, graphics and text includes many graphics drawing and painting tools sophisticated lighting effects, various image processing filters

Adobe Premiere provides many video and audio tracks, superimposition and

virtual clips supports various transitions, filters and motions for video clips a reasonable desktop video editing tool

93

Multimedia software tools

Multimedia authoring Microsoft Power Point

building slide show sequence slides include objects of different media, e.g., sound and video limited animation ability

Macromedia Director, Flash Movie metaphor (cast of bitmapped sprites, scripts, music,

sound, and palettes) Lingo script language allows more control of the presentation

sequence. Control of devices, e.g., VCRs and disk players. ready for building interactivities using buttons, etc.