Dalí A Multimedia Software

Library

Wei Tsang OoiBrian Smith, Hsi Haye Chan, Matthew Chiu, Sugata Mukhopadhyay, Dan Rubinovitz, Jiesang Song, Steve Weiss.

Motivations

Current Software Solution

BlackBox

C Source

MPEG to RealVideo

BlackBox

RealVideo

Encoder

RGB

MPEG Real

OpenFile(f)GetNextFrame()

MPEG to RealVideo

RGB

Decoder

YUV to RGB

Encoder

RGB to YUV

RealMPEG

YUV

compressed data

OpenFile(f)GetNextFrame()

YUV

MPEG to RealVideo

YUVDecoder

YUV to RGB

Encoder

RGB to YUV

RealMPEG

compressed data

Decode I-Frame only

BlackBox

RGB

MPEG

?OpenFile(f)GetNextFrame()

Decode I-Frame only

RGB

MPEG

?

Decoder

YUV to RGB

compressed data

YUV

OpenFile(f)GetNextFrame()

Decode I-Frame only

RGB

MPEG

?Decoder

YUV to RGB

I Frame filter

compressed data

I-frames

Others Motivations..

BlackBox

Read from network ? Tune I/O ? Run on handheld ?

Current Software Solution

BlackBox

C Source

Dalí is in between..

BlackBox

C Source

Dalí

Example

Example :

Process a subset of the gray scale version of each I-frame

MPEG structures

GOP hdrsequence hdr picture picture

pic hdr pic data

What we want to do

parse sequence header

What we want to do

parse sequence headerwhile not eof do

- find & parse picture header

What we want to do

parse sequence headerwhile not eof do

- find & parse picture header- if it is an I-frame then

- parse the picture into DCT domain

- IDCT into spatial domain

What we want to do

parse sequence headerwhile not eof do

- find & parse picture header- if it is an I-frame then

- parse the picture into DCT domain- IDCT into spatial domain- take the gray scale version of the frame and process it

Abstractions

Abstractions

For input/output For representing elements in MPEG For storing images in DCT domain For storing images in spatial

domain

BitStream & BitParser

BitStream

File

BitParser

BitParser

Parse Encode

MPEG Headers

MpegGopHdrMpegSeqHdr MpegPicHdrPicture Data

Find Skip Dump

Parse Encode

Find

MpegGopHdrMpegSeqHdr MpegPicHdrPicture Data

Find Skip Dump

Parse Encode

Skip

MpegGopHdrMpegSeqHdr MpegPicHdrPicture Data

Find Skip Dump

Parse Encode

Dump

MpegGopHdrMpegSeqHdr MpegPicHdrPicture Data

Find Skip Dump

MpegGopHdr

Parse Encode

Parse

MpegGopHdrMpegSeqHdr MpegPicHdrPicture Data

Find Skip Dump

width: 320height: 240pic rate: 30 fps etc

Parse Encode

Encode

MpegSeqHdr

Find Skip Dump

width: 320height: 240pic rate: 30 fps etc

ScImage

ScBlock :- { short dc; char index[63]; int ac[63];}

represents 8x8 pixels

ByteImage

unsigned char0..255

represents a pixel

The Code

Example

MpegSeqHdr *seqHdr = MpegSeqHdrNew ();MpegPicHdr *picHdr = MpegPicHdrNew ();BitStream *bs = BitStreamNew (65536);

BitStream

Example

BitParser *bp = BitParserNew ();BitParserAttach (bp, bs);

BitStream

BitParser

Example

BitStreamReadFromFile (bs, f, 0);

BitStream

BitParser File

MpegSeqHdr

MpegGopHdr

MpegPicHdr

Example

MpegSeqHdrFind (bp);

BitParser

MpegSeqHdr

MpegGopHdr

MpegPicHdr

Example

MpegSeqHdrParse (bp, seqHdr);w = (seqHdr->width + 7)/8;h = (seqHdr->height + 7)/8;

BitParser

MpegSeqHdr

MpegGopHdr

MpegPicHdr

ExampleScImage *y = ScNew (w, h);ScImage *u = ScNew (w/2, h/2);ScImage *v = ScNew (w/2, h/2);

y u v

Examplestatus = MpegPicHdrFind (bp);while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr);

status = MpegPicHdrFind (bp);}

Examplestatus = MpegPicHdrFind (bp);while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) {

}

status = MpegPicHdrFind (bp);}

Examplestatus = MpegPicHdrFind (bp);while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) { MpegPicIParse ( bp,seqHdr,picHdr, scy,scu,scv); : }

status = MpegPicHdrFind (bp);}

Examplestatus = MpegPicHdrFind (bp);while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) { MpegPicIParse ( bp,seqHdr,picHdr, scy,scu,scv); : } BitStreamFill(bp, f);

status = MpegPicHdrFind (bp);}

Refill the BitStream

BitParser

Refill the BitStream

BitParser

Refill the BitStream

BitParser File

Memory Mapped BitStream

bs = BitStreamMMapReadNew(“tennis.mpg”);

File

Operate on a subset

width: 8height: 6x: 0y: 0parentWidth: 8isVirtual: 0firstByte*

Physical ByteImage

width: 5height: 4x: 1y: 1parentWidth: 8isVirtual: 1firstByte*

Virtual ByteImage

Example

ScImage *virt = ScClip (y,6,2,20,20);

6 20 blocks

Example

ScImage *virt = ScClip (y,6,2,20,20);ByteImage *byte = ByteNew (160, 160);ScToByte (virt, byte);

6 20 blocks

Implementation

Implementation

C library Tcl bindings available

Organization

Code are organized into packages Basic

– ByteImage, ScImage, etc. MPEG

– MPEG-1 Video/Audio/System Vision

– Useful vision routines

Other Packages

AVI WAVE JPEG GIF

Availability

http://www.cs.cornell.edu/dali open source binaries for linux/win32/sunos examples documentations

The End