Post on 05-Dec-2014
description
transcript
Dalí A Multimedia Software
Library
Wei Tsang OoiBrian Smith, Hsi Haye Chan, Matthew Chiu, Sugata Mukhopadhyay, Dan Rubinovitz, Jiesang Song, Steve Weiss.
Motivations
Current Software Solution
BlackBox
C Source
MPEG to RealVideo
BlackBox
RealVideo
Encoder
RGB
MPEG Real
OpenFile(f)GetNextFrame()
MPEG to RealVideo
RGB
Decoder
YUV to RGB
Encoder
RGB to YUV
RealMPEG
YUV
compressed data
OpenFile(f)GetNextFrame()
YUV
MPEG to RealVideo
YUVDecoder
YUV to RGB
Encoder
RGB to YUV
RealMPEG
compressed data
Decode I-Frame only
BlackBox
RGB
MPEG
?OpenFile(f)GetNextFrame()
Decode I-Frame only
RGB
MPEG
?
Decoder
YUV to RGB
compressed data
YUV
OpenFile(f)GetNextFrame()
Decode I-Frame only
RGB
MPEG
?Decoder
YUV to RGB
I Frame filter
compressed data
I-frames
Others Motivations..
BlackBox
Read from network ? Tune I/O ? Run on handheld ?
Current Software Solution
BlackBox
C Source
Dalí is in between..
BlackBox
C Source
Dalí
Example
Example :
Process a subset of the gray scale version of each I-frame
MPEG structures
GOP hdrsequence hdr picture picture
pic hdr pic data
What we want to do
parse sequence header
What we want to do
parse sequence headerwhile not eof do
- find & parse picture header
What we want to do
parse sequence headerwhile not eof do
- find & parse picture header- if it is an I-frame then
- parse the picture into DCT domain
- IDCT into spatial domain
What we want to do
parse sequence headerwhile not eof do
- find & parse picture header- if it is an I-frame then
- parse the picture into DCT domain- IDCT into spatial domain- take the gray scale version of the frame and process it
Abstractions
Abstractions
For input/output For representing elements in MPEG For storing images in DCT domain For storing images in spatial
domain
BitStream & BitParser
BitStream
File
BitParser
BitParser
Parse Encode
MPEG Headers
MpegGopHdrMpegSeqHdr MpegPicHdrPicture Data
Find Skip Dump
Parse Encode
Find
MpegGopHdrMpegSeqHdr MpegPicHdrPicture Data
Find Skip Dump
Parse Encode
Skip
MpegGopHdrMpegSeqHdr MpegPicHdrPicture Data
Find Skip Dump
Parse Encode
Dump
MpegGopHdrMpegSeqHdr MpegPicHdrPicture Data
Find Skip Dump
MpegGopHdr
Parse Encode
Parse
MpegGopHdrMpegSeqHdr MpegPicHdrPicture Data
Find Skip Dump
width: 320height: 240pic rate: 30 fps etc
Parse Encode
Encode
MpegSeqHdr
Find Skip Dump
width: 320height: 240pic rate: 30 fps etc
ScImage
ScBlock :- { short dc; char index[63]; int ac[63];}
represents 8x8 pixels
ByteImage
unsigned char0..255
represents a pixel
The Code
Example
MpegSeqHdr *seqHdr = MpegSeqHdrNew ();MpegPicHdr *picHdr = MpegPicHdrNew ();BitStream *bs = BitStreamNew (65536);
BitStream
Example
BitParser *bp = BitParserNew ();BitParserAttach (bp, bs);
BitStream
BitParser
Example
BitStreamReadFromFile (bs, f, 0);
BitStream
BitParser File
MpegSeqHdr
MpegGopHdr
MpegPicHdr
Example
MpegSeqHdrFind (bp);
BitParser
MpegSeqHdr
MpegGopHdr
MpegPicHdr
Example
MpegSeqHdrParse (bp, seqHdr);w = (seqHdr->width + 7)/8;h = (seqHdr->height + 7)/8;
BitParser
MpegSeqHdr
MpegGopHdr
MpegPicHdr
ExampleScImage *y = ScNew (w, h);ScImage *u = ScNew (w/2, h/2);ScImage *v = ScNew (w/2, h/2);
y u v
Examplestatus = MpegPicHdrFind (bp);while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr);
status = MpegPicHdrFind (bp);}
Examplestatus = MpegPicHdrFind (bp);while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) {
}
status = MpegPicHdrFind (bp);}
Examplestatus = MpegPicHdrFind (bp);while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) { MpegPicIParse ( bp,seqHdr,picHdr, scy,scu,scv); : }
status = MpegPicHdrFind (bp);}
Examplestatus = MpegPicHdrFind (bp);while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) { MpegPicIParse ( bp,seqHdr,picHdr, scy,scu,scv); : } BitStreamFill(bp, f);
status = MpegPicHdrFind (bp);}
Refill the BitStream
BitParser
Refill the BitStream
BitParser
Refill the BitStream
BitParser File
Memory Mapped BitStream
bs = BitStreamMMapReadNew(“tennis.mpg”);
File
Operate on a subset
width: 8height: 6x: 0y: 0parentWidth: 8isVirtual: 0firstByte*
Physical ByteImage
width: 5height: 4x: 1y: 1parentWidth: 8isVirtual: 1firstByte*
Virtual ByteImage
Example
ScImage *virt = ScClip (y,6,2,20,20);
6 20 blocks
Example
ScImage *virt = ScClip (y,6,2,20,20);ByteImage *byte = ByteNew (160, 160);ScToByte (virt, byte);
6 20 blocks
Implementation
Implementation
C library Tcl bindings available
Organization
Code are organized into packages Basic
– ByteImage, ScImage, etc. MPEG
– MPEG-1 Video/Audio/System Vision
– Useful vision routines
Other Packages
AVI WAVE JPEG GIF
Availability
http://www.cs.cornell.edu/dali open source binaries for linux/win32/sunos examples documentations
The End