EE569 Digital Video Processing Copyright Xin Li 2007 1 Overview of Video Processing Video...

EE569 Digital Video Processing CopyriEE569 Digital Video Processing Copyright Xin Li 2007ght Xin Li 2007

11

Overview of Video ProcessingOverview of Video Processing

Video Acquisition

ComputerGraphics

Video Compression

VideoTransmission

VideoAnalysis

Video Manipulation Video

Display

ComputerVision

Video Database


22

Video AcquisitionVideo Acquisition

Video camera VHS digitization

computer-generated


33

Acquisition-related ProblemsAcquisition-related Problems

Video cameraVideo camera– What if camera is not kept still? (e.g., jittering in What if camera is not kept still? (e.g., jittering in

carphone sequence)carphone sequence)– Why is it difficult to improve the spatial resolution of Why is it difficult to improve the spatial resolution of

video cameras?video cameras?

VHS digitizationVHS digitization– What if VHS contains some scratches? What if VHS contains some scratches? – How to handle interlaced video?How to handle interlaced video?

Computer-generatedComputer-generated– How is this type of video different? Shouldn’t we have How is this type of video different? Shouldn’t we have

a separate coding algorithm for this type of video?a separate coding algorithm for this type of video?


44

High Frame Rate VideoHigh Frame Rate Video

Frame rate >>30fps

Applications: UAV, face detection/recognition, intelligent vehicle …

Machine vision can beat HVS


55

Video ManipulationVideo Manipulation

Why?Why?– Fight against a Fight against a non-idealnon-ideal video video

acquisition (e.g., analog heritage, film acquisition (e.g., analog heritage, film scratches, limited resolution) or scratches, limited resolution) or transmission environmenttransmission environment

– Create new and Create new and artificialartificial video content video content (e.g., spatio-temporal interpolation, (e.g., spatio-temporal interpolation, background/foreground modification)background/foreground modification)

video in, video out


66

Video DejitteringVideo Dejittering

http://epubs.siam.org/sam-bin/dbq/article/41869

PDE-based approach by Jackie Shen


77

Video InpaintingVideo Inpainting

Cool application: remove the annoying texts added byvarious video conversion software


88

Error ConcealmentError Concealment

some blocks are corrupteddue to channel errors

corrupted blocks are recovered From surrounding neighbors

in space and time


99

Deinterlacing Deinterlacing

field odd even odd even

interlaced

frame n n+1n-1

progressive

n-2


1010

SuperresolutionSuperresolution

… …

… …

LR sequence

HR sequence


1111

Post-processingPost-processingDeblocking: suppress block artifacts in video

decoded video frameat very low bit rate

processed video frameafter deblocking


1212

Video SynthesisVideo Synthesis

http://www-2.cs.cmu.edu/afs/cs/project/VirtualizedR/www/VirtualizedR.html


1313

Video MattingVideo Matting


1414

Video GamesVideo Games

Revenue: $7.1 billion in the US in 2005Revenue: $7.1 billion in the US in 2005


1515

Video DynamosaicsVideo Dynamosaics


1616

Dynamosaics ResultDynamosaics Result

Source: http://www.vision.huji.ac.il/dynmos/


1717

Video Coding OverviewVideo Coding Overview

The grand challengeThe grand challenge– We still face We still face insufficient storageinsufficient storage space for space for

video data even with Gigabyte hard disksvideo data even with Gigabyte hard disks– Video transmission through Video transmission through limited bandwidthlimited bandwidth

channels channels

Existing approachesExisting approaches– Three-dimensional waveform codingThree-dimensional waveform coding– Motion-compensated hybrid codingMotion-compensated hybrid coding– Model-based codingModel-based coding– Video coding standardsVideo coding standards


1818

Three-dimensional Waveform Three-dimensional Waveform Coding*Coding*

Image codingImage coding– Subband/wavelet coding of 2D signalsSubband/wavelet coding of 2D signals– Wavelet works because of its good Wavelet works because of its good

localization property in both space and localization property in both space and frequencyfrequency

From image to videoFrom image to video– 3D subband/wavelet coding3D subband/wavelet coding– Why direct extension does not work?Why direct extension does not work?– Recent advances: 2D+t vs. t+2DRecent advances: 2D+t vs. t+2D


1919

Motion-compensated Motion-compensated Predictive CodingPredictive Coding

Basic ideaBasic idea– DPCM coding in temporal domainDPCM coding in temporal domain– To reduce overhead on motion field, motion vector is To reduce overhead on motion field, motion vector is

assigned to each block instead of each pixelassigned to each block instead of each pixel– After block-wise motion compensation, code motion-After block-wise motion compensation, code motion-

compensated residues like still imagescompensated residues like still images

Variations: variable block size, fractional-pel Variations: variable block size, fractional-pel accuracy, overlapped block motion accuracy, overlapped block motion compensation (OBMC)compensation (OBMC)All existing video coding standards from H.261 All existing video coding standards from H.261 to the latest H.264 fall under such categoryto the latest H.264 fall under such category


2020

Model-based CodingModel-based Coding

Object-based codingObject-based coding– Attempt to replace Attempt to replace blocksblocks by by objectsobjects– Its success remains uncertain due to Its success remains uncertain due to

difficulty of segmentationdifficulty of segmentation

Knowledge-based codingKnowledge-based coding– Explicitly build 3D wireframe models to Explicitly build 3D wireframe models to

represent moving objectsrepresent moving objects– Limited successLimited success in videophone in videophone

applicationsapplications


2121

Video Coding StandardsVideo Coding Standards

ISO ITUMPEG-1 (1992)1.5Mbps, VCD

MPEG-2 (1996)2-10Mbps, DVD

MPEG-4 (2000)8-1024Kbps, videophone

Digital cinema (ongoing)

H.261 (1990)p×64Kbps

H.2638-64Kbps, videophone

H.263+/++8-64Kbps, videophone

H.264/AVC (2003)

windows media player(Microsoft)

real player(Real-Networks)Skype Video??


2222

Transcoding ProblemTranscoding Problem

How to translate a piece of MPEG2 (DVD) video How to translate a piece of MPEG2 (DVD) video into WMV format?into WMV format?– Straightforward approach: decode it by MPEG2 Straightforward approach: decode it by MPEG2

decoder and then encoder it by MSC MPEG4 encoderdecoder and then encoder it by MSC MPEG4 encoder– Transcoding approach: achieve the same goal with Transcoding approach: achieve the same goal with

reduced computational costreduced computational cost

When spatial or temporal resolution changes, When spatial or temporal resolution changes, the goal of complexity reduction becomes more the goal of complexity reduction becomes more difficult to achieve in transcodingdifficult to achieve in transcoding


2323

A Cautious Note about A Cautious Note about StandardsStandards

Why do we need standards?Why do we need standards?– Not us, but them (industry and business people)Not us, but them (industry and business people)– Basically it is a protocol or common language Basically it is a protocol or common language

representing the state of artrepresenting the state of art

What negative impact do standards have?What negative impact do standards have?– People tend to work on codec optimization instead of People tend to work on codec optimization instead of

asking fundamental questions (e.g., why does it asking fundamental questions (e.g., why does it work? How far is it from the theoretical bound?)work? How far is it from the theoretical bound?)

– You have to think out of the box!You have to think out of the box!


2424

New Directions in Video CodingNew Directions in Video Coding

Distributed video coding for Distributed video coding for sensor sensor networksnetworks– How to shift MC from encoder to decoder?How to shift MC from encoder to decoder?

Video coding for Video coding for cartooncartoon sequences sequences– Existing techniques work terribly on themExisting techniques work terribly on them

Video coding inspired by studies of Video coding inspired by studies of HVSHVS– You have seen the impact of motion maskingYou have seen the impact of motion masking– There also exists other properties of HVS that There also exists other properties of HVS that

can be exploited (e.g., foveation model) can be exploited (e.g., foveation model)


2525

Video Transmission (I)Video Transmission (I)Downloading Downloading – Pro: you can have your own copy and can Pro: you can have your own copy and can

watch it offlinewatch it offline– Con: you have to wait!!!Con: you have to wait!!!

StreamingStreaming– Pro: no need to store (we seldom watch a Pro: no need to store (we seldom watch a

movie again and again)movie again and again)– Con: you have to have a good network Con: you have to have a good network

connection and pray for less trafficconnection and pray for less trafficWhy does YouTube beat Google Video?


2626

Video Transmission (II)Video Transmission (II)

Multi-cast mode

6Mbps 3Mbps 1.5Mbps

6Mbps

3Mbps

1.5Mbps

Scalable mode


2727

Video Transmission (III)Video Transmission (III)

Description 1

Description 2

receiver 1

receiver 2

receiver 1+2

Exploit diversity to fight against adversary transmission environment


2828

Video Transmission Through NetworksVideo Transmission Through Networks

Networking protocolsNetworking protocols– Transmission Control Protocol (TCP)Transmission Control Protocol (TCP)– User Datagram Protocol (UDP)User Datagram Protocol (UDP)– Real Time Protocol (RTP) and VDPReal Time Protocol (RTP) and VDP– Real Time Streaming Protocol (RTSP)Real Time Streaming Protocol (RTSP)– ReSerVation Protocol (RSVP)ReSerVation Protocol (RSVP)

Transmission Control Protocol is not suitable for video Transmission Control Protocol is not suitable for video streamingstreaming because because– TCP imposes its own flow control and windowing TCP imposes its own flow control and windowing

schemes on the data stream, effectively destroying schemes on the data stream, effectively destroying temporal relations between video framestemporal relations between video frames

– Reliable message delivery is unnecessary for video - Reliable message delivery is unnecessary for video - losses are tolerable and TCP retransmission causes losses are tolerable and TCP retransmission causes further jitter and skew. further jitter and skew.


2929

Security issuesSecurity issues

Video is unique Video is unique – high data rate, power hungry, time high data rate, power hungry, time

constrained, loss-tolerant, content with constrained, loss-tolerant, content with varying importancevarying importance

Content access controlContent access control– Cryptographic approachesCryptographic approaches– Digital video scrambling techniquesDigital video scrambling techniques

Piracy and malicious attacksPiracy and malicious attacks– Video Video watermarkingwatermarking


3030

Video Content Protection by Video Content Protection by Watermarking TechniquesWatermarking Techniques

Signature insertion

Signature extraction


3131

Research IdeasResearch Ideas

Distributed video coding for error resilienceDistributed video coding for error resilience– Further extension of multiple descriptionsFurther extension of multiple descriptions– Motion estimation/compensation is performed at the Motion estimation/compensation is performed at the

decoder instead of encoder decoder instead of encoder

Power-constrained transmissionPower-constrained transmission– Sensor network applications and handheld devicesSensor network applications and handheld devices

Authentication in networked transmissionAuthentication in networked transmission– Transmission errors vs. malicious attacksTransmission errors vs. malicious attacks– Transcoding distortions vs. intentional attacksTranscoding distortions vs. intentional attacks


3232

Video AnalysisVideo Analysis

Motion segmentationMotion segmentation– In contrast to image segmentation, motion In contrast to image segmentation, motion

offers valuable clues for separating different offers valuable clues for separating different objectsobjects

Motion trackingMotion tracking– Track the same object across video framesTrack the same object across video frames

Motion interpretationMotion interpretation– Easy for HVS, difficult for a computer (e.g., Easy for HVS, difficult for a computer (e.g.,

summarize a 6-hr. baseball video into 30min.)summarize a 6-hr. baseball video into 30min.)


3333

Motion SegmentationMotion Segmentation


3434

Motion TrackingMotion Tracking


3535

Motion InterpretationMotion Interpretation

Scene change detectionScene change detection– Where motion tracking failsWhere motion tracking fails

Cut, dissolve, wipe classificationCut, dissolve, wipe classification– Those are artificial features added by video Those are artificial features added by video

editing staffediting staff

Analyze each video segmentAnalyze each video segment– Camera motion: panning or zooming or stillCamera motion: panning or zooming or still– Object motion: shape, direction, speed, etc.Object motion: shape, direction, speed, etc.


3636

Extract “important” motion pictures such as home-runs

Application (I): Video Application (I): Video SummarizationSummarization


3737

Application in swimming pool monitoring to prevent drowning

Application (II): Video-based Application (II): Video-based LifeguardLifeguard


3838

Application (III): Irregularity Application (III): Irregularity DetectionDetection

Source: http://www.wisdom.weizmann.ac.il/~vision/Irregularities.html


3939

Video Database ManagementVideo Database Management

Database managementDatabase management– Indexing, parsing, browsing, queryingIndexing, parsing, browsing, querying– RetrievalRetrieval

What is special about video?What is special about video?– Formidable amount of dataFormidable amount of data– Difficulty with query (Difficulty with query (contentcontent-based) -based) – Inherent uncertainty and imprecisionInherent uncertainty and imprecision


4040

If you want to join Google …If you want to join Google …


4141

Content-Based Video Retrieval Content-Based Video Retrieval (CBVR)(CBVR)

How to provide a How to provide a compact compact and and complete complete video sequence representation?video sequence representation?– Spatial analysis (histogram, color, texture)Spatial analysis (histogram, color, texture)– Temporal analysis (cut, dissolve, wipe)Temporal analysis (cut, dissolve, wipe)

How to provide How to provide easy-to-useeasy-to-use and and efficientefficient query interface to userquery interface to user– Video browsing (slide vs. 3D)Video browsing (slide vs. 3D)– Video querying (example-based, text-based)Video querying (example-based, text-based)


4242

Compressed-domainCompressed-domain Video Analysis Video Analysis

Since video data often exist in compressed Since video data often exist in compressed format, it is preferred to do analysis with bit format, it is preferred to do analysis with bit streams rather than pixel valuesstreams rather than pixel values– Examples: caption detection, shot detection etc.Examples: caption detection, shot detection etc.

The key issue lies in how to exploit the The key issue lies in how to exploit the information contained in the bit streaminformation contained in the bit stream– It does not cost much computation It does not cost much computation – It is constrained by the adopted compression It is constrained by the adopted compression

techniques and never perfect (e.g., block motion field)techniques and never perfect (e.g., block motion field)


4343

Course OutlineCourse OutlineVideo formation, perception and representation

2D ME techniques

video coding

new trend

streaming

denoising, deinterlacing,deblocking

super-resolution

error control andconcealment

motion segmentation

motion tracking

motion interpretation

Textbooks have partially covered the highlighted topics

Date post:	19-Dec-2015
Category:	Documents
Upload:	chastity-daniel
View:	222 times
Download:	2 times

EE569 Digital Video Processing Copyright Xin Li 2007 1 Overview of Video Processing Video...

Documents