41570904 Remote Media Immersion

5/24/2018 41570904 Remote Media Immersion

1/17

2010

Remote

Medi a

Imm

ersion

The Remote Media Immersion (RMI) system is the resultof a unique blend of multiple cutting-edge mediatechnologies to create the ultimate digital mediadelivery platform. The main goal is to provide animmersive user experience of the highest quality. RMI

encompasses all end-to-end aspects from mediaacquisition storage transmission up to their finalrendering. !pecifically the "ima streaming mediaserver delivers multiple high band#idth streamstransmission error and flo# control protocols ensuredata integrity and high-definition video combined #ithimmersive audio provide highest quality rendering. TheRMI system is operational and has been successfullydemonstrated in small and large venues. Relying on thecontinued advances in electronics integration andresidential broadband improvement RMI demonstratesthe future of on-demand home entertainment


2/17

ABSTRACT

The Remote Media Immersion (RMI) system is the result of a uniqueblend of multiple cutting-edge media technologies to create the ultimate digital

media delivery platform. The main goal is to provide an immersive user

experience of the highest quality. RMI encompasses all end-to-end aspects

from media acquisition storage transmission up to their final rendering.

!pecifically the "ima streaming media server delivers multiple high

band#idth streams transmission error and flo# control protocols ensure data

integrity and high-definition video combined #ith immersive audio provide

highest quality rendering. The RMI system is operational and has been

successfully demonstrated in small and large venues. Relying on the continuedadvances in electronics integration and residential broadband improvement

RMI demonstrates the future of on-demand home entertainment.

$lectronics $ngineering !eminar Topic % Instrumentation !eminar


3/17

INTRODUCTION

The charter of the Integrated Media !ystems &enter (IM!&) at the

'niversity of !outhern &alifornia ('!&) is to investigate ne# methods and

technologies that combine multiple modalities into highly effective immersivetechnologies applications and environments. ne of the results of these

research efforts is the Remote Media Immersion (RMI) system. The goal of the

RMI is to create and develop a complete aural and visual environment that

places a participant or group of participants in a virtual space #here they can

experience events that occurred in different physical locations. RMI technology

can effectively overcome the barriers of time and space to enable on demand

the realistic recreation of visual and aural cues recorded in #idely separated

locations.

The focus of the RMI effort is to enable the most realistic recreation of

an event possible #hile streaming the data over the Internet. Therefore #e

push the technological boundaries much beyond #hat current video-on-demand

or streaming media systems can deliver. s a consequence high-end rendering

equipment and significant transmission band#idth are required. The RMI

pro*ect integrates several technologies that are the result of research efforts at

IM!&. The current operational version is based on four ma*or components that

are responsible for the acquisition storage transmission and rendering of high

quality media.

STAGES OF RMI



4/17

Acquisition of high-quality !"ia st#!as

This authoring component is an important part of the overall chain to

ensure the high quality of the rendering result as experienced by users at a later

time. s the saying +garbage in garbage out, implies no amount of qualitycontrol in later stages of the delivery chain can mae up for poorly acquired

media.

R!al-ti! "igital sto#ag! an" $lay%ac& of ulti$l! in"!$!n"!nt st#!as

"ima !calable !treaming Media rchitecture provides real-time

storage retrieval and transmission capabilities. The "ima server is based on a

scalable cluster design. $ach cluster node is an off-the-shelf personal computer

#ith attached storage devices and for example a ast or /igabit $thernet

connection. The "ima server soft#are manages the storage and

net#or resources to provide real-time service to the multiple clients that are

requesting media streams. Media types include but are not limited to M0$/-1

at 2T!& and 34T5 resolutions multichannel audio (e.g. 67.1 channel

immersive audio) and M0$/-8.

'#otocols fo# synch#oni(!") !ffici!nt #!alti! t#ansission of ulti$l!

!"ia st#!as

selective data retransmission scheme improves playbac quality #hilemaintaining realtime properties. flo# control component reduces net#or

traffic variability and enables streams of various characteristics to be

synchroni9ed at the rendering location. Industry standard net#oring protocols

such as Real-Time 0rotocol (RT0) and Real-Time !treaming 0rotocol (RT!0)

provide compatibility #ith commercial systems.

R!n"!#ing of i!#si*! au"io an" high #!solution *i"!o

Immersive audio is a technique developed at IM!& for capturing the

audio environment at a remote site and accurately reproducing the completeaudio sensation and ambience at the client location #ith full fidelity dynamic

range and directionality for a group of listeners (6: channels of uncompressed

linear 0&M at a data rate of up to 6;.:Mb Mb


5/17

si9es may range from a single person or family at home to a large venue seating

hundreds. or the visual streams #e decided that #e required at least high-

definition (34) resolution as defined by T!&6. The highest quality T!&

modes are either 6?17 @ 67=7 pixels at an interlaced frame rate of 1?.?; per

second or 61=7 @ ;17 pixels at a progressive frame rate of >?.?8 per second.or the audio rendering #e rely on the immersive audio technology developed

at IM!& #hich utili9es a 67.1 channel playbac system. The rendering

capabilities of immersive audio goes much beyond current stereo and >.6

channel systems. The combination of 67.1 channels of immersive audio and

high-definition video is the next step in audio-visual

fidelity. $ach presentation session retrieves and plays bac at least one high-

definition visual and one immersive aural stream in synchroni9ation. 2ote that

this choice #as imposed by the available media content and is not an inherent

limitation in the "ima design. The streams are stored separately on the server

for t#o reasons. irst the RMI system is designed to be extensible such that

additional video or other streams may become part of a presentation in the

future. !econd allo#ing streams to be separately stored enables RMI to

retrieve different components of a presentation from different server locations.

The final fine-grained synchroni9ation is achieved at the client side. The on-

demand delivery of the streams that form anRMI presentation is enabled

through our streaming media architecture called "ima. Aith features such as

scalability multi-stream synchroni9ation transmission error and flo# control it

is uniquely suited for RMI-style media delivery.

DEI+ER. OF /IG/-RESOUTION MEDIA



6/17

n important component of delivering isochronous multimedia over I0

net#ors to end users and applications is the careful design of a multimedia

storage server. The tas of such a server is t#ofoldB (6) it needs to efficiently

store the data and (1) it must schedule the retrieval and delivery of the data

precisely before it is transmitted over the net#or. RMI relies on our "imastreaming media architecture. "ima has been designed from the ground up to

be a scalablemedia delivery platform that can support multiple very high

band#idth streams.

igure sho#s the server cluster architecture #hich is designed to harness

the resources of many nodes and many dis drives per node concurrently.

ur current implementation the server consists of a 8- #ay cluster of rac-

mountable 0&s running Red 3at Cinux ho#ever larger configurations arepossible to increase the number of concurrent RMI sessions supported. $ach

cluster node is attached to a local net#or s#itch #ith a ast or /igabit

$thernet lin. The nodes communicate #ith each other and send the media data

via these net#or connections. The local s#itch is connected to both a A2

bacbone (to serve distant clients) and a C2 environment #ith

local clients. &hoosing an I0 based net#or eeps the per-port equipment cost

lo# and is immediately compatible #ith the public Internet. The media data is

stored in segments (called blocs) across multiple say four high performance

hard dis drives. There are t#o basic techniques to assign the data blocs of amedia ob*ect D in a load balanced manner D to the magnetic dis drives that

form the storage systemB in a round-robin sequence or in a random manner.



7/17

"ima uses a pseudo-random data placement in combination #ith a deadline-

driven scheduling approach.

This combination enables short startup latencies and can easily support

multimedia applications #ith non sequential data access patterns including

variable bit rate (5ER) video or audio and interactive operations such pauseand resume etc. It also enables efficient data reorgani9ation during system and

storage scaling. or the delivery of media data #e tae full advantage of this

high performance storage layout.

E##o# Cont#ol 0ith S!l!cti*! R!t#ansissions f#o Multi$l! S!#*!# No"!s

ne of the characteristics of continuous media streams is that they require datato be delivered from the server to a client location at a predetermined rate. This

rate may vary over time for streams that have been compressed #ith a variable

bit rate (5ER) media encoder. 5ER streams enhance the rendering quality

ho#ever they #ill generate bursty traffic on a pacet s#itched net#or such as

the Internet. This in turn can easily lead to pacet loss due to congestion. !uch

data loss adversely affects compressed audio

and video streams because much of the temporal or spatial redundancy in the

data has already been removed by the compression algorithm. urthermore

important data such as audio


8/17

the transmission bet#een the server and a rendering location. The "ima cluster

architecture taes advantage not only of the distributed storage resources

among the multiple nodes but also of the multiple net#or connections that

lin all the nodes together. Media data is transmitted via the real-time protocol

(RT0) encapsulated in connection-less '40 datagrams. To avoid trafficbottlenecs each node transmits the data blocs that it holds directly to the

clients via RT0. 3ence each client #ill receive RT0 data pacets fromeach

server node#ithin the cluster. 2ote that the current Internet infrastructure #as

not designed for streaming media and provides only best-effort pacet delivery.

The RMI test environment. server is located at the Information !ciences Institute in rlington 5

#hile the rendering is performed in Cos ngeles.

Therefore RT0


9/17

source I0 address the client can easily establish the complete set of server

nodes. nce a server receives a request it checs #hether it holds the pacet

and either ignores the request or performs a retransmission. &onsequently this

approach #astes net#or band#idth and increases server load.

Unicast R!t#ansissions The second more efficient and scalable method of

sending retransmission requests requires that the unique server node that holds

the missing pacet be identified. This could be accomplished in several #ays.

or example the client could reproduce the pseudo-random number sequence

that #as originally used to place the data cross multiple server nodes. This

approach has several dra#bacs. irst identical algorithms on both the clients

and the servers must be used at all times. If

the server soft#are is upgraded then all clients must be upgraded immediately

too. The logistics of such an undertaing can be daunting if the clients are

distributed among thousands of end users. !econd during

scaling operations the number of server nodes or dis drives changes and hence

ne# parameters need to be propagated to the clients immediately. ther#ise

the server nodes #ill be misidentified. Third if for any reason the client

computation is ahead or behind the server computation (e.g. the total number

of pacets received does not match the number of pacets sent) then any future

computations #ill be #rong. This could potentially happen if the client has only

a limited memory and pacets arrive sufficiently out-of-sequence. more

robust approach is as follo#s. The client determines the server node from

#hich a lost RT0 pacet #as intended to be delivered by detecting gaps innode-specific pacet sequence numbers. Ae term these local sequence numbers

(C!2) as opposed to theglobal sequence number (/!2) that orders all pacets.

lthough this approach requires pacets to contain a node-specific sequence

number along #ith a global sequence number the clients require very little

computation to identify and locate missing pacets.

Cli!nt-S!#*!# A"a$ti*! Flo0 Cont#ol

RMI relies on the efficient transfer of multimedia streams bet#een aserver and a client. These media streams are captured and displayed at a

predetermined rate. or example video streams may require a rate of 18 1?.?;

G7 >?.?8 or :7 frames per second. udio streams may require 88677 or

8=777 samples per second. n important measure of quality for such

multimedia communications is the precisely timed playbac of the streams at

the client location. chieving this precise playbac is complicated by the

popular use of variable bit rate (5ER) media stream compression. 5ER

encoding algorithms allocate more bits per time to complex parts of a stream

and fe#er bits to simple parts to eep the visual and aural quality at nearconstant levels. or example an action sequence in a movie may require more

bits per second than the credits that are displayed at the end. s a result

different transmission rates may be required over the length of a media stream



10/17

to avoid starvation or overflo# of the client buffer. s a contradictory

requirement #e #ould lie to minimi9e the variability of the data transmitted

through a net#or. 3igh variability produces uneven resource utili9ation and

may lead to congestion and exacerbate display disruptions. The focus of the

RMI flo# control mechanism is on achieving high quality media playbac byreducing the variability of the transmitted data and hence avoiding display

disruptions due to data starvation or overflo# at the client. Ae designed a novel

technique that ad*usts the multimedia traffic based on an end to end rate control

mechanism in con*unction #ith an intelligent buffer management scheme.

'nlie previous approaches #e consider multiple signaling thresholds and

adaptively predict the future band#idth requirements. Aith this Multi-

Threshold Flow Control (MT&) scheme 5ER streams are accommodated

#ithout a priori no#ledge of the stream bit rate. urthermore because the

MT& algorithm encompasses server net#or and clients it adapts itself to

changing net#or conditions. 4isplay disruptions are minimi9ed even #ith fe#

client resources (e.gB a small buffer si9e). The MT& protocol is currently

implemented in our "ima streaming media server and clients. This allo#s us to

measure and verify its effectiveness in an end-to-end application such as RMI.

The design of a high performance rate control algorithm #as motivated in part

by our continuous media server implementation. Ae required an adaptive

technique that could #or in a real-#orld dynamic environment #ith minimal

prior no#ledge of the multimedia streams to be served. Ae identified the

follo#ing desirable characteristics for the algorithmB

Onlin! o$!#ationB Required for live streaming and also desirable for storedstreams.

Cont!nt in"!$!n"!nc!B n algorithm that is not tied to any particularencoding technique #ill continue to #or #hen ne# compression algorithms

are introduced.

Minii(ing f!!"%ac& cont#ol signalingB The overhead of online

signaling should be negligible to compete #ith offline methods that do not needany signaling.

Rat! soothingB The pea data rate as #ell as the number of rate changesshould be lo#ered compared #ith the original unsmoothed stream. This #ill

greatly simplify the design of efficient real-time storage retrieval and transport

mechanisms to achieve high resource utili9ation. &onsidering these ob*ectives

#e designed our novel Multi-Threshold lo# &ontrol (MT&) technique.



11/17

It distinguishes itself from previously proposed algorithms by incorporating the

follo#ingB (6) a multi-threshold buffer model (1) a consumption prediction

component and (G) a modular rate change computation frame#or in #hich

ne# consumption prediction algorithms and feedbac delay estimation

algorithms can easily be incorporated.

RENDERING

The rendering side of the RMI system is composed of several parts. The video

and audio streams are received over a sufficiently fast I0 net#or connection

on a personal computer running t#o instances of the "ima playbac soft#are.

This media player is structured into several components and only one of them

interfaces #ith the actual media decoder. This allo#s us to plug-in multiple



12/17

soft#are and hard#are decoders and hence support various media types. or

RMI one of the players interfaces #ith a &ine &ast

34 M0$/-1 decompression board manufactured by 5ela Research. This

decoder accepts M0$/-1 compressed high definition video at data rates in

excess of >7Mb7 Mb


13/17

deliver sound over loudspeaers these methods also require precise

cancellation of the crosstal signals resulting from the opposite side

loudspeaer in order to deliver the desired sound to each ear. s a result they

#or #ell only for a single listener in a precisely-defined position.

Multichannel surround sound systems have also been developed that use threefront channels and t#o surround channels to provide a sense of envelopment

for multiple listeners. lthough these systems may #or #ell in accompanying

movies they are not suitable for immersive systems. The main reason is that

they leave significant spatial gaps in the a9imuth plane (e.g. at ?7 degrees to the

side of the listeners) and provide no coverage in the median plane (no elevation

cues). In order to minimi9e locali9ation errors the number of loudspeaers must

increase linearly #ith the #idth of the listening area. listener that moves *ust

a fe# cm from the designated listening spot is sub*ected to high imaging

distortion and no longer experiences the correct locali9ation cues. This problem

can be addressed by increasing the number of loudspeaers. It is no#n from

the psychoacoustics literature that human listeners can locali9e sounds very

precisely in the front hemisphere and less precisely to the sides and in the rear

hemisphere.

Therefore locali9ation errors from the front channels that arise from offsets in

the listener position from the desired center location #ill be particularly

evident. In our implementation #e allocate five channels to the front hori9ontal

plane by augmenting the traditional three front loudspeaers #ith t#o

additional #ide channels. The advantages of this are t#ofoldB (i) locali9ation

errors in the front hemisphere are reduced and (ii) the #ide channels providesimulated side-#all reflection cues that have been sho#n to increase the sense

of spatial envelopment. In addition to the > front channels a rear surround

channel is added to fill the gap directly behind the listener. $levation

information is reproduced from t#o height channels placed above the front left

and right loudspeaers. $arly experiments that #e have performed #ith this

configuration have sho#n that these 67 channels significantly increase the

sense of locali9ation and envelopment for listeners. ma*or challenge in multi-

listener environments arises from room acoustical modes particularly in small

rooms that cause a significant variation in the responses measured at differentlistener locations. Responses measured only a fe# cm apart can vary by up to

6>-17 dE at certain frequencies. This maes it difficult to equali9e an audio

system for multiple

simultaneous listeners. urthermore it maes it impossible to render a remote

performance for a large audience and have them experience the sound exactly

as it is being produced at the remote event. 0revious

methods for room equali9ation have utili9ed multiple microphones and spatial

averaging #ith equal #eighting. This approach tends to smooth out large

variations due to standing #aves but does not tae into account the effects ofroom modes that for some frequencies may be more concentrated in one region

and not in another. Ae have developed a ne# method that derives a



14/17

representative room response from several room responses that share similar

characteristics. This response can then be used to equali9e the entire class of

responses it represents.

DISTRIBUTED IMMERSI+E 'ERFORMANCE

DEMONSTRATION

ur current #or is a real-time multi-site interactive specific reali9ation of the

immersive environment called 4istributed Immersive 0erformance (4I0). The

Remote Media Immersion experiments and demonstrations primarily utili9ed

uni-directional transmissions and off-line audio and video processing. The 4I0

pro*ect leverages the RMI technology and extends its capabilities to multi-directional multi-participant and real-time interaction in synchronous and

collaborative music performance. The goal of the 4I0 pro*ect is to develop the

technology for performances in #hich the participants - subsets of musicians

the conductor and the audience - are in different physical locations and are

interconnected by high fidelity multichannel audio and video lins as sho#n in

igure.

There are generally t#o classes of participants #ith different sets of

ob*ectives and requirements. Ae label the participants acti*! or $assi*!depending on their level of interaction and the latency they can tolerate. orexample in a tele-conference application all participants are generally active.

or a performance event there is a mix of active participants (musicians in a

concert players in a sports event panelists at a meeting #hose primary actions

are those of doing of physically engaging in the activity) and passive

participants (the audience #hose primary actions are

seeing and listening).



15/17

The Remote Media Immersion system is the next step in audio-visual

fidelity for streaming media delivered on-demand over the Internet. The goal of

the RMI is to push the boundaries beyond #hat is currently available in any

commercial system or other research prototype. Its emphasis is on the highest

quality of audio-visual experiences and most realistic immersive rendering. Toachieve goal #e #ere faced #ith a number of unique challenges and had to

design novel techniques to mae RMI a reality. In this report #e presented the

error and flo# control algorithms as #ell as the immersive audio aspects of

RMI. The current RMI setup is out of reach for most home users. or

#idespread adoption a number of technological advances #ill be necessary

#hich subsequently #ill lead to more affordable prices and mae the RMI

system feasible for high-end home use. or example #e are currently using the

M0$/- 1 algorithm at a lo# compression ratio to achieve our target visual

quality. n improvement in compression algorithms and affordable hard#are

availability #ill most certainly mae the same quality available at lo#er bit

rates in the future (M0$/-8 is a candidate here). 3ence #e envision a cross-

over point in the next fe# years #hen the band#idth required for RMI is belo#

the bit rates that ne# high-speed residential broadband technologies can

deliver (for example very high-speed 4!C or 54!C). In the more distant

future one might envision a portable device #ith a high resolution screen and a

personal immersive audio system. or a single listener it is possible to achieve

very enveloping audio rendered #ith only t#o speaers as long as the position

and shape of the listenerFs ears are no#n. The 04of the future #ill at some

point have enough processing capabilities to combine visual tracing #ithadaptive rendering of both audio and video.

CONCUSION

The Remote Media Immersion system is the next step in audio-visual fidelityfor streaming media delivered on-demand over the Internet. The goal of the

RMI is to push the boundaries beyond #hat is currently available in any

commercial system or other research prototype. Its emphasis is on the highest

quality of audio-visual experiences and most realistic immersive

rendering. To achieve our goal #e #ere faced #ith a number of unique

challenges and had to design novel techniques to mae RMI a reality. In this

report #e presented the error and flo# control algorithms as #ell as the

immersive audio aspects of RMI. The current RMI setup is out of reach for

most home users. or #idespread adoption a number of technological advances#ill be necessary #hich subsequently #ill lead to more affordable prices and

mae the RMI system feasible for high-end home use. or example #e are



16/17

currently using the M0$/- 1 algorithm at a lo# compression ratio to achieve

our target visual quality. n improvement in compression algorithms and

affordable hard#are availability #ill most certainly mae the same quality

available at lo#er bit rates in the future (M0$/-8 is a candidate here). 3ence

#e envision a cross-over point in the next fe# years #hen the band#idthrequired for RMI is belo# the bit rates that ne# high-speed residential

broadband technologies can deliver (for example very high-speed 4!C or

54!C). dditionally #e are #oring to#ards simplifying and automating the

setup and calibration procedures that are currently necessary. !ubsequently

these signal processing algorithms can be incorporated into multichannel home

receivers at moderate cost. In the more distant future one might envision a

portable device #ith a high resolution screen and a personal immersive audio

system. or a single listener it is possible to achieve very enveloping audio

rendered #ith only t#o speaers as long as the position and shape of the

listenerFs ears are no#n. The 04 of the future #ill at some point have

enough processing capabilities to combine visual tracing #ith adaptive

rendering of both audio and video.

REFERENCES

&oncert 3all coustics ". ndo !pringer 5erlag 2e# "or

Hournal of the coustical !ociety of merica

&onference on Multimedia &omputing and 2et#oring (MM&2)

!anta &lara &alifornia

R. immermann J. u &. !hahabi !.-". 4. "ao and 3. hu. "imaB

4esign and $valuation of a !treaming Media !ystem for Residential

Eroadband !ervices

###.ho#stuff#ors.com

http://www.howstuffworks.com/http://www.howstuffworks.com/


17/17

###.docstoc.com

http://www.docstoc.com/http://www.docstoc.com/

Date post:	14-Oct-2015
Category:	Documents
Upload:	shreejith-nair
View:	12 times
Download:	0 times

41570904 Remote Media Immersion

Documents