University of Illinois at Chicago Electronic Visualization Laboratory (EVL) Global Scale...

University of Illinois at ChicagoElectronic Visualization Laboratory (EVL)

Global Scale Tele-Immersion Network Performance Activities

Jason Leigh, Oliver Yu, Linda Winkler, Alan Verlo, Tom DeFanti

Yong-joo Cho, Ray Fang, Javier Girado, Liujia Hu, Tomoko Imai, Naveen Krishnaprasad, Michael Lewis, Ya Ju Lin, Dave Pape, Kyoung Park, Chris Scharver,

Brenda Silva, Liang WangJosh Eliason, Jinghua Ge, Eric He, Atul Nayak, Shalini Venkatamaran


Common Characteristics of Teleimmersive Applications

CAVE

Im m ersaDesk

Tele-Im m ersionClients

Tele-Im m ersionServer

Rem ote Data &Com putation

Services

Compute or DatabaseQuery Spawned by

Tele-ImmersionClient and M anaged

by Tele-ImmersionServer


Characterization of Tele-Immersive Streams

Estimated bandwidth

(bits/s)

DiffServ

Types BurstinessLatency sensitive

Jitter sensitive

Error sensitive

UDP avatar 6K x n

(15fps)

Interactive Real-time

Constant Y Y N

UDP audio stream

64K x n Brief Y Y N

UDP video stream

10M

(2-way only)Constant Y Y YN

UDP stream

With Playback dependsNon-

interactive Real-time

Constant Y N YN

TCP control data 7K x n Reliable Brief YN YN Y

TCP bulk datadepends

Best Effort or Deadline Delivery

Sustained burst

N N Y


Network Research


Maximizing Bandwidth Utilization over Long Fat Networks

• Even if QoS via DiffServ or IntServ is available, it still does not solve the Long Fat Network problem

• Problem is small TCP window sizes (well known problem but still no widely accepted solution)

• On SGI’s change in window size requires kernel rebuild

• Size of window should be set to current available BW of the network

• CAVERNsoft’s Parallel Socket Striping works well but is considered “irresponsible” use of networks


64K Window SizeAmsterdam to Chicago

Achieved bandwidth while transmitting 50M from Amsterdam to Chicago over a 45Mbps link

0

10

20

30

40

1 3 5 7 9 11

13

15

17

19

21

23

25

27

29

Number of Sockets

Ba

nd

wid

th (

Mb

ps)

Bursty as max bw reachedbut performance is still good


64K Window SizeCERN to EVL

Plot of Average Achievable Bandwidth vs # of Parallel TCP Sockets Used to Deliver a 50M File from Switzerland (CERN) to Chicago over a 45Mbps link

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

# of Sockets

Ban

dwid

th (M

bps)

Bursty as max bw reachedbut performance still good


Window Size: EVL = 1.85M, SARA = 64KPlot of Average Achievable Bandwidth vs # of Paralle TCP Sockets Used to Deliver a 50M

File from Chicago to Amsterdam over 100Mbps Link

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

# of Sockets

Ba

nd

wid

th (

Mb

ps)

Plot of Average Achievable Bandwidth vs # of Paralle TCP Sockets Used to Deliver a 50M File from Amsterdam to Chicago over 100Mbps Link

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

# of Sockets

Ba

nd

wid

th (

Mb

ps)

SARA to EVL

EVL to SARA

Sending client determines the window size

When window size is large enoughno real benefit to using parallel sockets


Window size: EVL = 1.85M, CERN = 640KPlot of Average Achievable Bandwidth vs # of Paralle TCP Sockets Used to Deliver a 50M

File from Chicago to Switzerland (CERN) over 45Mbps Link

0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

# of Sockets

Ba

nd

wid

th (

Mb

ps)

Plot of Average Achievable Bandwidth vs # of Paralle TCP Sockets Used to Deliver a 50M File from Switzerland (CERN) to Chicago over 45Mbps Link

0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

# of Sockets

Ba

nd

wid

th (

Mb

ps)

CERN to EVL

EVL to CERN

Similar storyat CERN


Anomalies

• Theoretical BW from EVL to SARA is 100Mbps• Netperf UDP shows reasonable performance:

– EVL to SARA 85Mbps– SARA to EVL 65Mbps (5 more hops via Abilene)

• Netperf and Parallel sockets TCP shows only:– 30Mbps

• Perhaps due to asymmetric tcp window size settings?

• Argument for UDP-based schemes?E.g. Forward Error Correction


Forward Error Correction scheme for low-latency delivery of error sensitive data

• Transmit error correction data over high bandwidth networks that can be used for correcting UDP streams to achieve lower latency than TCP but higher reliability.

• Transmit error correction data to improve quality of streamed video by correcting for lost packets.

• Not intended for bulk data transfer but in light of TCP results this might hold some promise.


FEC Experiments

• EVL to SARA- Amsterdam (45Mb/s 100ms RT latency)

• Broader Ques:– Can FEC provide a benefit? How much?– Tradeoff between redundancy and benefit?

• Specific Ques:– TCP vs UDP vs FEC/UDP– How much jitter does FEC introduce?– High thru put UDP vs FEC/UDP to observe loss &

recovery


`Latency of transmitting 100 packets underUDP, TCP, FEC/UDP with 3:1 redundancy.

0

50

100

150

200

250

300

350

400

0 500 1000 1500 2000 2500

Packet size in bytes

1-w

ay la

ten

cy in

ms

UDP

TCP

FEC over UDP

FEC greatest benefit is in small packets.

Larger packets impose greater overhead.

As redundancy decreases FEC approaches UDP.

goal


Jitter for UDP, TCP and FEC over UDPMoving average (over 20 successive data points) of deviations of Short Term Latency (also over 20

successive data points)

0

2

4

6

8

10

12

14

1 5 9

13

17

21

25

29

33

37

41

45

49

53

57

61

65

69

73

77

Jit

ter

UDP

TCP

FEC/UDP

G o a l


Packet Loss over UDP vs FEC/UDP between Chicago & Amsterdam

Data Rate(bits/s)

Packet Size(Bytes)

Packet Loss Rate in UDP (%)

Packet Loss Rate in FEC over UDP (%)

1M 128 0.4 0

1M 256 0.2 0

1M 1024 0.2 0

10M 128 30 4

10M 256 25 3

10M 1024 21 1.5

UDP

UDP

FEC


Human Factors in Tele-Immersion


Collaborative Coordination Experiments between Chicago and Singapore

CAVE to CAVE (STAR TAP)• Audio via Phone call• Scramnet (adjustable latency, 0 jitter)• LAN Ethernet (~ 10ms)• Local ISDN (~ 200ms)• STAR TAP (~ 250ms)• Predict STAR TAP similar to performance over ISDN


Collaborative Coordination Experiments between Chicago and Singapore

0

1

2

3

4

5

6

7

8

9

Scramnet 10ms LAN Ethernet ~10ms Scramnet 200ms ISDN ~200ms

Completion Time (s)

Collisions

• 200ms RTT is the threshold where performance begins to suffer• Roughly RTT to Asia. Results to Singapore similar to local ISDN

200ms RTT with 0 jitter is same as 10ms RTT with 7ms jitter


DiffServ Bandwidth

0

5

10

15

20

25

30

1 15

29

43

57

71

85

99

113

127

141

155

169

183

197

211

225

239

Time (s)

Ba

nd

wid

th (

Mb

ps)

DiffServ Latency

0

10

20

3040

50

60

70

1 15

29

43

57

71

85

99

113

127

141

155

169

183

197

211

225

239

Time (s)

1 w

ay L

ate

ncy (

ms)

DiffServ Packet Loss

0

100

200

300

400

500

1 15

29

43

57

71

85

99

113

127

141

155

169

183

197

211

225

239

Time (s)

Pa

cke

t L

oss (

pa

cke

ts/s

)

25Mbps80Mbps

ANL

EVL

42Mbps 42Mbps

100Mbps 100Mbps

100Mbps 100Mbps

fore back

Bandwidth recovery good

Latency recovery good

Small packet loss

DiffServ Experiment 1+ background + DiffServ

x

x

xx


DiffServ Experiment 2DiffServ Bandwidth

0

5

10

15

20

25

1 11

21

31

41

51

61

71

81

91

101

111

121

131

141

151

161

Time (s)

Ba

nd

wid

th (

Mb

ps)

DiffServ Latency

0

50

100

150

200

250

300

1 11

21

31

41

51

61

71

81

91

101

111

121

131

141

151

161

Time (s)

1 w

ay L

ate

ncy (

ms)

DiffServ Packet Loss

0200400600800

1000120014001600

1 10

19

28

37

46

55

64

73

82

91

100

109

118

127

136

145

154

163

Time (s)

Pa

cke

t L

oss

(p

acke

ts/s

)

25Mbps80Mbps

ANL

EVL

42Mbps 42Mbps

100Mbps 100Mbps

100Mbps 100Mbps

fore

back

Bandwidth recovery good

Latency recovery not good

Packet loss double

+ background + DiffServ

x

x

x x


Application of Research Results

CAVERNsoft G2

applications at iGrid 2000 in Yokohama


Tele-Immersion MiddlewareThe CAVERNsoft G2 Toolkit

• G2 is C++ toolkit for building Tele-Immersive applications with special emphasis on networking

• Networking:– UDP, TCP, Multicast, HTTP.– UDP reflector and multicast bridge.– TCP reflector.– Remote procedure calls.– 32 and 64bit Remote file I/O.– Parallel 32 & 64 bit TCP socket striping for high throughput data delivery.– FEC library.– Client/Server distributed shared memory persistent database.– Threading, Mutual Exclusion.– Built-in Instrumentation of networking services.– QoS via GARA and MCSP underway.


Tele-Immersion MiddlewareThe CAVERNsoft G2 Toolkit

• Audio streaming.

• Articulated Avatars.

• VR navigation.

• VR menus.

• Speech recognition with IBM ViaVoice.

• Collaborative application shell to jumpstart development.


TIDE

• Teleimmersive Data Explorer (TIDE)

• In collaboration with National Center for Data Mining

• General framework for collaborative visualization of massive data-sets

• Current data-set is ozone data from NOAA


CIBRView

• Collaborative Image Based Rendering Viewer(CIBRview)

• In collaboration with Wes Bethel and Steve Lau at Lawrence Berkeley Lab

• Accesses volume data 512x256x256x 256 frames ~ 40Gig data-sets

• Generates image slices that are distributed to collaborating clients.

• Sent about 500 slices/files from Chicago to Japan


Virtual HarlemUniversity of Missouri

Virtual Harlem

Reconstruction of Harlem during the

Harlem Renaissance 1920-40


Earthquake HypocentersSpace Physics & Aeronomy Research Collaboratory

(U of Michigan)A demonstration at Telecom 2000 and SC 2000

between Israel, Dallas, Chicago, Michigan


Network Visualization


QoS Internet Monitoring ToolQoSIMoto


STAR TAP Network Visualization


Future Work

• DiffServ and RSVP from EVL to CERN in collaboration with NWU

• Reliable UDP for high throughput bulk data transmission• Integrated Collaboratory for Analysing Networks (iCAN):

iCAN-Monitor, iCAN-Visualize, iCAN-Manage, iCAN-Active Test, iCAN-Collaborate

Date post:	27-Mar-2015
Category:	Documents
Upload:	melanie-quinn
View:	216 times
Download:	3 times

University of Illinois at Chicago Electronic Visualization Laboratory (EVL) Global Scale...

Documents