Comp 294-9 :: Fall 2003

Video As A Datatype

Ketan Mayer-Patel

Comp 294-9 :: Fall 2003

The mothers of all video...• NTSC - American standard• PAL - European standard• SECAM - French standard (ugly stepchild)• For better or worse, this is where it all starts.

Comp 294-9 :: Fall 2003

Color Spaces• One luminance component• Two chrominance components

– Original TV was black and white.– Adding color had to be done in a compatible way.

• NTSC: YIQ• PAL: YUV• In general: YUV and YCrCb used as terms.

Comp 294-9 :: Fall 2003

YUV vs. RGB• Simple linear transform from one cube to

the other.• Specific transform coefficients vary for

specific systems, but a common one:– Y = 0.299 R + 0.587 G + 0.114 B– Cr = -0.147 R - 0.289 G + 0.437 B + 0.5– Cb = 0.615 R - 0.515 G - 0.100 B + 0.5

Comp 294-9 :: Fall 2003

YUV Challenges• YUV is like taking RGB cube, standing it

on a corner with (0,0,0) on bottom and (1,1,1) at top, rotating slightly, and taking the bounding box.

• What problems might this incur?– Some RGB colors are illegal.– A lot of YUV colors are illegal.

Comp 294-9 :: Fall 2003

A Note On Color• Previous view of color is EXTREMELY

simplified.– Color is complicated

• Frequency dependent response for contrast, lightness, etc.

– Gamma correction.`

Comp 294-9 :: Fall 2003

Scanning• Analog video signal is continuous voltage

signal that gets scanned along the screen.• The electron stream controlled by two

orthogonal sets of magnets.– Horizontal: Beam is moved from left to right

and then quickly back.– Vertically: Beam is moved from top to bottom

and then quickly back.

Comp 294-9 :: Fall 2003

Scanning illustrated

Time

Vol

tage

Comp 294-9 :: Fall 2003

Structure of Video• Blinn’s article.

– Macrostructure• Frequency peaks at frame rate.

– Microstructure• Frequency peaks at line rate

• Adding color is a good first example of source-aware channel coding.– What was the problem?– What was the solution?– Why does it work?– What are its drawbacks (i.e., when does it fail)?

Comp 294-9 :: Fall 2003

Interlacing• Progressive = every scan line done in turn.• Interlaced = every other scan line done.

– Creates two fields: odd field and even field.– NTSC: 262.5 lines per field at 60 fields per sec.– PAL: 312.5 lines per field at 50 fields per sec.

• Fields are separated in time.

Comp 294-9 :: Fall 2003

Analog Bandwidths• True meaning of the bandwidth.• Y, U, and V signals are all continuous along

a scan line.– A bit of a hybrid signal: discrete vertically,

continuous horizontally.• NTSC:

– Y is 4.2 MHz wide, I is 2 MHz, Q is 1 MHz• PAL:

– Y is 6 MHz wide, U is 3 MHz, V is 3 MHz

Comp 294-9 :: Fall 2003

Corresponding Data Rates• How much data can you put in 6 MHz band?

– Depends on S/N ratio.– Depends on modulation scheme.

• Typical: 27 - 36 Mbs• How many cable channels do you get?

– If 50, then 1.3 - 1.8 Gbs coming into your home.• The real question for multimedia is:

– Why haven’t we found a better use for 1.3 Gbs than continuous broadcast of Real World.

Comp 294-9 :: Fall 2003

Digital Video Frames• Almost always progressive• 3 planes of pixel values (Y, U, and V)• Pixel depth• Geometry of each plane: width x height

– Chrominance is generally subsampled.• How the planes relate to each other.• Frame rate.

Comp 294-9 :: Fall 2003

CCIR-601• Standard established for digitizing NTSC

and PAL signals.DigitalComponent

AnalogCounterpart

CCIR-601Recommendation

Height Lines NTSC: 480PAL: 576

Width Continuousvoltage changesalong scan line.

Sample at 13.5 MHzResults in 702 pixelsRecommended 720

Pixel depth Voltage range. Gamma-corrected8-bit sample.

Chrominancesubsampling

Bandwidthdifferences

4:2:2, 4:2:0, 4:1:1

Comp 294-9 :: Fall 2003

Why 8-bits for chrominance?• What’s another way to cut chrominance

bandwidth in half?– Use 4-bits per pixel.

• Why won’t that work?– Need the dynamic range for color.

• But what about when Y is either small or large?– Don’t need the range, but lots of YUV combinations

that won’t ever be used.

Comp 294-9 :: Fall 2003

4:2:2• For every 4 luminance samples, take 2

chrominance samples from odd lines and 2 from even lines.

• Chrom. planes just as tall, half as wide.• JPEG does this.

Comp 294-9 :: Fall 2003

4:2:0• 2 chrominance samples for every 4

luminance samples, odd lines only.• Chrominance halved in both directions.• MPEG generally does this.

Comp 294-9 :: Fall 2003

4:1:1• What should this be?

– 1 chrominance for every 4 luminance for both odd and even lines.

– And that is what it is.• But sometimes used to refer to this:

Comp 294-9 :: Fall 2003

De-interlacing• Since analog video fields are separated in

time by 1/2 the frame rate, at least half of a digital frame is missing no matter where you sample from.

• Normal solution: linear interpolation of even fields to produce matching samples for odd fields.

• Even better: linear interpolation of both fields to produce matching samples and thus digital frame rate will equal field rate.

Comp 294-9 :: Fall 2003

Film Frame Rates• Film is a different beast altogether.

– Continuous both vertically and horizontally– Projection is simultaneous for all parts of the

picture. – Expensive medium. – Combination of all of this motivates 24 fps.

• Film to digital is easier than video to digital.– No interlacing, sample where you want to.

• Film to video is harder.

Comp 294-9 :: Fall 2003

3:2 Pulldown• Converts 24 frames to 60 fields.

1 2 3 4 5

1e 1o 2e 2o 3e 2o 3e 3o 4e 4o 5e 5o

Comp 294-9 :: Fall 2003

Overall sampling lesson• Can’t recover what you don’t have.• Conversion between representations

requires estimation of missing samples.• Interpolation causes errors:

– Spatially: at the edges.– Temporally: when moving.

Comp 294-9 :: Fall 2003

Common Digital Video Sizes• CCIR-601 720x480 4:2:2, 4:2:0• SIF 360x240 4:2:0• CIF 360x288 4:2:0• 4:3 HDTV 1440x1152 4:2:2, 4:2:0• 9:16 HDTV 1920x1152 4:2:2, 4:2:0• 4CIF, 16CIF, QCIF

Comp 294-9 :: Fall 2003

Digital Bitrates• Current television:

– 30 fps * 720 * 480 * 1.5 * 8 = 124 Mb/s• 9:16 HDTV

– 30 fps * 1920 * 1152 * 1.5 * 8 = 796Mb/s• This motivates compression.