110420Cloud_Gamingfor_ECE498

8/12/2019 110420Cloud_Gamingfor_ECE498

1/34

Cloud Gaming on Mobile Devices

Presented by: Shu Shi

ECE498HP, Spring 2011

Wednesday, April 20, 2011


2/34

About Me

4thyear Ph.D. in CSDepartment

Advisers: Prof. Roy Campbell,Prof. Klara Nahrstedt

Research Interests:multimedia systems,multimedia networking,mobile computing

Publications: MM09, MM10,ICME10, ICME11, ...



3/34

Cloud Gaming

Onlive: www.onlive.com What is Cloud Gaming?

http://www.youtube.com/watch?v=Fr2DSxmrH4Y

3

http://www.offlinecarrot.com/blog/52/onlive-in-australiadetailsis-onlive-coming-to-australia

http://www.offlinecarrot.com/blog/52/onlive-in-australiadetailsis-onlive-coming-to-australiahttp://www.offlinecarrot.com/blog/52/onlive-in-australiadetailsis-onlive-coming-to-australiahttp://www.youtube.com/watch?v=Fr2DSxmrH4Yhttp://www.youtube.com/watch?v=Fr2DSxmrH4Yhttp://www.onlive.com/http://www.onlive.com/


4/34

Remote Rendering

4

RenderingServer

ClientSource

Contents

RenderingResults

User

InteractionRequests

DisplayNetwork

User

Interaction

Save

Computation

Depends on high-bandwidth low-

latency network



5/34


6/34


7/34

Mobile Cloud Gaming

7

RenderingServer

SourceContents

RenderingResults

UserInteractionRequests

Display

UserInteraction

Wireless

Network

Why is mobile cloud gaming difficult?

How to solve the problems?

How to evaluate the proposed solutions?Wednesday, April 20, 2011


8/34

Experiment Platforms

BZFlag: http://bzflag.org/ An open source 3D tank shooting game

Video

http://www.youtube.com/watch?v=t7Me1RDJIXw

8

RenderingServer

Mobile

Client

Wireless

Networks

http://www.youtube.com/watch?v=t7Me1RDJIXwhttp://www.youtube.com/watch?v=t7Me1RDJIXwhttp://bzflag.org/http://bzflag.org/


9/34

Interaction Latency

9

Mobile

Devices

Rendering

ServerGame

Engine

Game Scene Frame

F2

F1

F3

Image Frame

After Rendering

I1

I2

I3.

.

....

.

.

.

.

.

.

Fi

User Changes the

Rendering Viewpoint Server Updates the

Rendering Viewpoint

First Frame Rendered

at the Updated

Rendering ViewpointFirst Image Frame

at the UpdatedRendering Viewpoint

Interaction

LatencyIi



10/34

Interaction Latency

10

Viewpoint Update

RenderingServer

SourceContents

RenderingResults

Display

UserInteraction

Wireless

Network

Interaction latency depends on the round trip timeof wireless networks

The latency of wireless networks, especially cellular networksare very high (UMTS 200+ms, EDGE 300+ms, GPRS 700+ms)

High latency affects gaming experience 100 ms is the largest tolerable latency for the first person

shooting game



11/34

Interaction Latency Reduction

11

Viewpoint Update

RenderingServer

SourceContents

RenderingResults

Display

UserInteraction

Wireless

Network

Can wen reduce the interaction latency from thered path to the green path?



12/34

Remote Rendering Approaches

12

RenderingServer

ClientNetwork

Model

Forwarding

Point-based

Rendering

Model

Filtering

Image

Streaming

Environment

Map

3D

Warping

Double

Warping

Network Bandwidth and Computation on Clientmore less



13/34

Environment Map

Server sends environment maps to client e.g., panoramic map Supported by OpenGL or omni-directional camera

Advantage: eliminate the interaction latency

Disadvantage: only support pan & tilt

13QuickTime VRAn Image-Based Approach to Virtual Environment Navigation

http://www.cs.berkeley.edu/~ravir/6998/papers/p29-chen.pdfhttp://www.cs.berkeley.edu/~ravir/6998/papers/p29-chen.pdfhttp://www.cs.berkeley.edu/~ravir/6998/papers/p29-chen.pdf


14/34

3D Image Warping

14

F2

F1

F3I1

I2

I3.

.

....

.

.

.

.

.

.

User Changes the

Rendering Viewpoint

to Vnew

(Vnew.pos = V.pos)

Fi

Ei+V

E1+V

E2+V

E3+V

Environment Map

Viewpoint

Ii_new =

Env_proj(Ei,V,Vnew)

.

.

.

.

.

.

...

.

.

.

Fj

Ej+Vnew

Mobile

Devices

Rendering

ServerGame

Engine

Game Scene FrameImage Frame

After Rendering



15/34

3D Image Warping

Server sends depth images to client Image Map: I Depth Map: D

Advantages: eliminate the latency for any interaction

Disadvantages: artifacts (holes) are generated

15



16/34

3D Image Warping

16



17/34

3D Image Warping

17

F2

F1

F3I1

I2

I3.

.

....

.

.

.

.

.

.User Changes the

Rendering Viewpoint

to Vnew

Fi

Ii+Di+V

I1+D1+V

I2+D2+V

I3+D3+V

Image Map

Depth Map

Viewpoint

Ii_new =

Warping(Ii,Di,V,Vnew)

.

.

.

.

.

.

...

.

.

.

Fj

Ij+Dj+Vnew

Mobile

Devices

Rendering

ServerGame

Engine

Game Scene FrameImage Frame

After Rendering



18/34

Double Warping

Server sends two depth images per frame to client

18



19/34

Double Warping

19



20/34

Double Warping

20



21/34

Double Warping

21



22/34

Double Warping

22



23/34

Double Warping

Server sends two depth images per frame to client Advantages: eliminate interaction latency and

reduce warping artifacts

Disadvantages: more bandwidth to transmit moredepth images

23



24/34

v2

v1

v2

v1

v+

Reference Selection Problem

Select two or more virtual camera viewpoints togenerate a reference frame (depth image set) thatminimizes the warping error in a warping-basedremote rendering system

24

Rendering

Server

Mobile

Client

3D Video Frame

Reference Frame

Warping Result Rendering Result

- < Terr



25/34

DOL= 10 log10

MAX2I

PL D(Ivm, Iv

+

m ) +

n1

x=m

tx D(Ix, Iv+

x )

Performance Evaluation

Distortion Over Latency Latency is not enough to

evaluate system

performance

Warping quality is alsoimportant when userinteraction happens

Integrate both latency andquality into one score

(DOL)

25

tmtime

Iv

m+1Iv

m I

v

n1 I

v+

n

v+

Iv

m

Iv+

m I

v+

m+1 Iv+

n1

... ...ActuallyReceived

...

Im Im+1 In1 I

v+

n

tn-1tm+1PL

UL

Displaywith

Image Warping

......

...tmv

+

Iv

mIm+1

In1... Iv+ntn-1tm+1

Displaywithout

Image Warping

......

...UL

...

v+

Iv

mIv+

n

IdealDisplay

... ......



26/34

Performance Evaluation

26

SourceMedia

Content

User Interaction

v+

Viewpoint Manager

Rendering Engine

Sn Iv+

n

...In1Im+1

...

Rendering EngineSource

History Buffer Image WarpingResult

History Buffer

DOL

n

distortion

m, n

...

Image Warping

Display

ViewpointManager

Timer

v+

v+v+

v+m

v+

, m

PL, tx

Sn+1

Sn Sm+1 Sm Iv+

m I

v

m

Im

Iv

n1... Iv

m+1

Rendering Server Mobile Client

PL, tx

Im

Im

Run-time DOL Monitor Calculate Distortion Over Latency on the rendering server

at run-time

Rendering server runs the same image warping operationsas mobile client to calculate the warping quality



27/34

Results

27

Test Scenarios Simulator: running on laptop using 802.11n Wi-Fi Wi-Fi: running on iPhone 4 using 802.11g Wi-Fi 3G: running on iPhone 4 using AT&T 3G

Test Approaches Image Stream: send image frames only Single Warping: send one depth image for 3D image warping Double Warping: send two depth images for double warping



28/34

Results

28

Test Sequence Bunny: static 3D model Taichi: dynamic 3D video

Test Results

Interaction Latency: the less the better Update Latency: for reference DOL: the larger the better



29/34

Results

29


Test Results




30/34

Results

30


Test Results




31/34

Summary

Concepts of cloud gaming Using cloud server to render 3D video games Using low-latency broadband networks to stream

Major challenges of mobile cloud gaming Interaction latency

One possible solution to reducing latency Generate reference frames for local image warping

Performance evaluation for new approaches

Consider both latency and quality 31Wednesday, April 20, 2011


32/34

Questions?

32



33/34

Double Warping in

Mobile Cloud Gaming

33

v1 v2 v3



34/34

Challenges of Mobile

Cloud Gaming

Bandwidth High-quality video stream: 640x480, 30fps Raw video bandwidth: 640x480x12x30 = 100+Mbps Limited bandwidth (1-2Mbps) available from wireless

mobile networks

Compression ratio: 100:1

34

0 10 20 30 40 50 600

1

2

3

4

5

6

7

8

9

10

Time (s)

BitRate(M

bps)

DownlinkMean

Date post:	03-Jun-2018
Category:	Documents
Upload:	ngo-minh-tri
View:	217 times
Download:	0 times

110420Cloud_Gamingfor_ECE498

Documents