Real-Time Architectures 2003/2004johanl/educ/2IN20/Large slides/RTA... · TU/e Informatica, System...

Reinder J. Bril, [email protected]/e Informatica, System Architecture and Networking

1

Real-Time Architectures2003/2004

A real-time architecture formultimedia consumer terminals

(MCTs)

Reinder J. Bril03-05-2004


2

A real-time architecture for MCTs• Multimedia Consumer Terminals (MCTs):

– Trends;– Characteristics.

• Media processing in software:– A Digital Video Platform;– Scalable Video Applications;– Challenges & approaches.

• A co-operative QoS approach:– Multi-disciplinary team;– System.

• A retrospect on real-time issues


3

Multimedia Consumer Terminals

DVD CDxfront end

YC interface

IEEE 1394interface

DVB Tuner

Cable modem

CVBSinterface

VGA

RF Tuner Focus:Receivers in broad-cast environments

High-quality video applications


4

Trends• Digitization and compression:

– DVD, DTV

• Featurization:– Software services:

• TXT, OSD, menus, habit learning.– Processing:

• Sharpness enhancement, 100 Hz, PiP, dual screen.

• Convergence of services:– Delayed viewing (TV + VR/DVD-recorder);– Games (TV + game-console);– TV-commerce and interaction (TV + internet).

• Upgrading:– New standards for D-STB via satellite.


5

Characteristics

• High quality:– Including timeliness of audio and video.– Example: video enhancement.

Original (24 Hz)

Up-scaled (60 Hz)

Input stream: movie

Rendered stream: TV screen

QoS trade-off:Lesser picture quality often better than temporal incorrectness.


6

Characteristics

• High dependability:– Must work, immediately, and at all times.

• High Volume Electronics:– Global market with multiple standards;– Short product development cycles;– Product families.

• Low power:– No fan in a TV set …

• High pressure on price:– DVD recorder from $2000 to $200 in 2 years.


7

Traditional High-End TV Architecture

TXT CPUacq control

progROMRAM

PICNICenhancement

PALplusIC16:9 helper

MEM MEM

HFelectr.

standarddecoder

PICNICenhancement

picturecontrol

audiodemod.

Audioproces.

cableantenna

PAL/NTSC

display

YUV1fh

YUV2fh

RGB2fh

picturecontrol

standarddecoder

PiP+

Mem RGB2fh

CVBS

MEM

100HzFALCON

IC

MEM

MEM

NICAMdecod.demod

MPEGaudio

MPEGvideo

MEM

chan.decod.

transp.demux

channelbits

Traditional TVs, Set-Top Boxes:

• Fixed algorithms for fixed HW architectures

• Upgrade for new services/applications is problematic

• Systems are not flexible


8

Media processing in software

• Based on programmable platforms• Advantages:

– Upgrading;– Fast time-to-market of new features;– Scalable video applications → product families.

• Boundary conditions:– Cost-effectiveness;– Preserve existing qualities.


9

Media processing in software

• A Digital Video Platform• Scalable video applications• Challenges & approaches


10

A Digital Video Platform

Control + Services

Media processing

Bus

VLIWCPU

MIPSCPU

Memory

Coprocessors


11

HW Architectures vs SW Applications

Low-

end

Mid-

range

High-

end

Algorithm1 m

in max

Algorithm3

Algorithm2

Algorithm4

Resources

ProductFamilies

SW-Modules

Algorithm1

Algorithm2

Algorithm4

Algorithm3


12

Traditional vs Scalable Approach

Cost

Functionalitytarget limit

traditionalsystems

scalableapproach

Functionality

Quality

target limit

traditionalsystems

scalableapproach


13

Example: One High-Quality Application


14

HQ Video-Conference, Reduced Background Quality


15

Quality Levels in Algorithms


16

Scalable Algorithm

FUNCTION 1 FUNCTION 2

FUNCTION 4

FUNCTION 3

QUALITY CONTROL

control signal for

quality level

signal in

signal out

ALGORITHM FOR MEDIA PROCESSING

Quality

Resource needs


17

SVAs: Output Quality Mapping

Bandwidth CPU cyclesMemoryCo-processor cycles

Resources

Quality Levels1l 2l 3l 4l

13R

23R

12R

11R

14R

1l 2l 3l 4lQuality Levels

Output quality


18

Challenges & approaches• Maximize system utility, given available resources:

– Elements (semantically neutral):• Utility of individual applications;• Resource usage;• Relative importance.

– Approach:• static: [Lee et al 99].• dynamic:

– Applications: local quality control;– System: global system utility control.

• Resource sharing:– Interference between applications

• Predictability;• System robustness

– Approach: resource reservation.


19

Challenges & approaches

• Mode changes:– Examples:

• System:– Start of new application: main → main + pip;– Change of user focus: main → pip;– Exchange of main and pip.

• Application:– movie (24 Hz film) to commercial (50 Hz camera);– Channel switching.

– Smooth transitions required– Approach: mode-change protocol(s).


20

Example: Main → Main + PiP

dmux

audiodec.

dec. sharpenh.main: scalable

DVD

mixer

mixer :non-scalable

audiorend

read

scaler

digitizer: non-scalable

digit

pip: scalable

(1) Initial: Main

(2) Intermediate: reduce quality

(3) Intermediate: start PiP(4) Final: Main + PiP


21

Mode change protocol

User focus change: Main →PiP

main

PiPQuality

Qualitydecrease

main

PiPBudget

timeBudget

configuration change

Qualityincrease


22

Example: Video enhancement modesVideo Enhancement - Main Window

Video Enhancement - PiP

AnArtRAnArtR

1D scaling

1D scaling

Error Conceal.

Error Conceal. 2D Scaling

2D Scaling Color / Gamma

Color / Gamma

DigArtRDigArtR

Scan-RateConvers.

Scan-RateConvers.

DigArtRDigArtR

2D Scaling

2D ScalingError

Conceal.

ErrorConceal.

DigArtRDigArtR

480p 16:9

Scan-RateConvers.

Scan-RateConvers.

Scan-RateConvers.

Scan-RateConvers. Color /

Gamma

Color / Gamma

Color / Gamma

Color / Gamma

ATSC 720p/16:9

ATSC 1080i/16:9

NTSC 480i/4:3

DVD 480p/16:9

2D scaling

2D scalingError

Conceal.

ErrorConceal.

2Dscaling

2Dscaling

Error Conceal.


2D Scaling 2D

Scaling

2D Scaling

2Dscaling

2Dscaling

PiPColor / Gamma

Color / Gamma

Color / Gamma

Color / Gamma

Color / Gamma

Color / Gamma

2D Scaling

2D Scaling

NTSC 480i/4:3ATSC 1080i/16:9ATSC 720p/16:9DVD 480p/16:9

Note: Number and order of algorithms change for Main Window upon input change !


23

Example: Video enhancement modesInitial situation

AnArtRAnArtR

NTSC 480i/4:3 Main

PiPError Conceal.


2D Scaling 2D

Scaling

2D Scaling

Color / Gamma

Color / Gamma

ATSC 1080i/16:9

Scan-RateConvers.

Scan-RateConvers. 1D

scaling

1D scaling

Exchange of Main and PiP

Final situation

Error Conceal.


2D Scaling Color / Gamma

Color / Gamma

DigArtRDigArtR

Scan-RateConvers.

Scan-RateConvers.

ATSC 1080i/16:9

2Dscaling

2DscalingNTSC

480i/4:3PiP

Main


24

Challenges: dynamic load

time

load

structural load

running averagetemporal load

“worst-case” load

Close-to-averageresource allocation

MPEG decoding of DVD stream


25

A co-operative QoS approach

• Organization (multi-disciplinary):– Video domain experts;– System software specialists;– Mathematicians.

• System:– Adaptive applications:

• Scalable video applications;• Local quality control.

– QoS-based resource management (QRM):• Resource reservations;• Global system utility control.


26

Co-operative QoS approach: system

Adaptive applications

Provide quality levels + estimated resource req.

Resource manager Provides guaranteed resource budgets

Local quality control

SVAs…

Global system utility controlOptimizes system utility, sets quality levels + allocates resources


27

QoS approach: local quality control

• Synchronous (3D graphics):– Estimate `work’ for next frame and set operational

quality level according to available resources.• Asynchronous (high quality video):

– Through buffering and work-ahead;– Progress (i.e. amount of budget [= virtual time] till

deadline);– Quality selection based on penalties & reward:

• Deadline miss (−);• Quality level fluctuation (−);• Quality level (+).


28

Co-operative QoS approach• Shared responsibilities:

– QRM: enabling;– Applications: executing.

• Cost-effectiveness:– QRM:

• resource allocation according to relevance;• Low system overhead;• Close to average-case resource allocation.

– Applications: effective and efficient media processing.• Robustness:

– QRM: resource reservations;– Applications: “get by” with budget.

• Seamless switching between modes:– QRM: mode-change protocol;– Applications: smooth transitions.


29

A retrospect on real-time issues• Real-time is a QoS parameter !

– Deadline miss: a penalty for local quality control.• Modes of a system:

– Number of applications (main, pip, disk);– Modes of applications (input, output);– Quality settings and budgets of applications.

• Hierarchies:– Control:

• Global system utility control;• Local quality control (of an RCE).

– Scheduling:• Periodic budgets;• Asynchronous tasks (sharing a budget).


30

References• [Bril et al 01] R.J. Bril, C. Hentschel, E.F.M. Steffens, M.

Gabrani, G.C. van Loo, and J.H.A. Gelissen, Multimedia QoS in consumer terminals (invited lecture), In: Proc. IEEE Workshop on Signal Processing Systems (SIPS), pp. 332 – 343, September 2001.

• [Lee et al 99] C. Lee, J. Lehoczky, R. Rajkumar, and D. Siewiorek, On quality of service optimization with discrete QoS options, In: proc. 5th IEEE Real-Time Technology and Applications Symposium (RTAS), pp. 276 – 286, 1999.

• [Steffens et al 03] L. Steffens, S. van Loo, and C. Otero Perez, Trends in Operating Systems – Resource Management for Future CE Systems (invited lecture), ESTIMEDIA Workshop, http://www.idt.mdh.se/estimedia/steffens.pdf, October 2003.

• [Wüst et al 04] C.C. Wüst and W.F.J. Verhaegh, Quality control for Scalable Media Processing Applications, Journal of Scheduling, 7(2): 105-117, March 2004.

http://www.idt.mdh.se/estimedia/steffens.pdf

Real-Time Architectures2003/2004A real-time architecture for MCTsMultimedia Consumer TerminalsTrendsCharacteristicsCharacteristicsTraditional High-End TV ArchitectureMedia processing in softwareMedia processing in softwareA Digital Video PlatformHW Architectures vs SW ApplicationsTraditional vs Scalable ApproachExample: One High-Quality ApplicationHQ Video-Conference, Reduced Background QualityQuality Levels in AlgorithmsScalable AlgorithmSVAs: Output Quality MappingChallenges & approachesChallenges & approachesExample: Main Main + PiPMode change protocolExample: Video enhancement modesExample: Video enhancement modesChallenges: dynamic loadA co-operative QoS approachCo-operative QoS approach: systemQoS approach: local quality controlCo-operative QoS approachA retrospect on real-time issuesReferences

Date post:	09-Feb-2021
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Real-Time Architectures 2003/2004johanl/educ/2IN20/Large slides/RTA... · TU/e Informatica, System...

Documents