INDOT Communication Experience with Satellite Internet Service and SiteManager

Indiana DOT

Jim NugentJim SynderDeb Hood

Purdue University

Darcy BullockJim KrogmeierTC Chen

Outline

SiteManager Software ApplicationNeed for Satellite IP ServiceTachyon Satellite PerformanceSilverback Satellite PerformanceDial up PerformanceConcluding remarks about latency and chatty protocols!

SiteManager

12 States in Production, 9 additional licensed (total 21 states).SiteManager has online and offline operating modes.Indiana is currently in rollout phase Material testing, payments, administration.

Indiana relies extensively on field offices and wanted to evaluate satellite IP performance with SiteManager (and supporting applications)

INDOT Construction Management Practices

Decentralized modelField offices (trailers) are integral part of construction jobs.

IP Issues

Many field office are in remote areas with out land line broadband access.Satellite broadband access has the potential to make field trailer deployment a well defined bid item.

Objective

Investigate the suitability of satellite IP service for INDOT’s data communication between remote field offices and INDOT headquarter.Evaluate how well SiteManger, Outlook, and other applications perform with a satellite based broad band connection.

Test Equipment

Location: A trailer at Division of Research, West Lafayette, IN

Tachyon Outdoor Unit (ODU) Tachyon Indoor Unit (IDU)

Network Topology

AMC-4

Tachyon ODU

VPN router

VPN over Internet

VPN over Internet

Tachyon IDU(10.13.200.1)

Test Laptop with IRIS(10.13.200.7)

Remote Office(West Lafayette)

TSG

Tachyon Backbone Switch

VPN router

Tachyon(San Diego)

INDOT HQ(Indianapolis)

SiteManager Server(10.13.24.67)

Outlook Exchange Server(10.13.20.14)

Propagation Delay

Request for time slot

Actual data transmission

540 ms

+ 640 ms

1180 ms !!

One-way Delay

Internet

Internet

Remote Office(West Lafayette)

Tachyon(San Diego)

INDOT HQ(Indianapolis)

135 ms 135 ms ~50 ms

Selected Test Applications

Microsoft OutlookMicrosoft Windows Network DriveSiteManagerFTP

Tachyon Scenarios

T1 at Division of ResearchTachyon Satellite Service QoS 384kbps (downlink) and 384kbps

(uplink) Normal mode QoS 384kbps (downlink) and 384kbps

(uplink) with ALOHA QoS 1.5Mbps (downlink) and 512kbps (uplink)

with ALOHANormal mode – Uplink is reservation-based without

contentionALOHA mode – A contention-based algorithm where data less than 200 bytes is transmitted without a scheduled slot. If it collides, system falls back to reservation scheme. Performs best when uplink link is idle and packets small.

Test Procedure

Use IRIS (www.eeye.com) and Bandwidth Monitor Pro (www.bandwidthmonitorpro.com) to log the traffic for each task in each scenario (see Appendix for a list of tasks)Performance metric - response timeEach task is repeated 5 times except for some very long ones

T1(1.544Mbps) Task group I clock time

Task group 1: short tasks (under 25 seconds)

0

5

10

15

20

25

30

Clo

ck t

ime

(sec

onds

)

O.01

O.02

O.05

O.06

O.07

O.08

E.01

E.02

SM

.01

SM

.02

SM

.03

SM

.05

SM

.07

SM

.08

SM

.09

SM

.10

SM

.11

SM

.12

SM

.13

SM

.14Performance at Division of Research (task group 1)

T1 Time

T1 Results Task group II clock time

Task group 2: long tasks (over 25 seconds)

4585

5280

1430

0

50

100

150

200

250

300

350

400

450

500

Clo

ck t

ime

(sec

onds

)

O.03

O.04

E.03

E.04

E.05

E.06

FT

P.01

FT

P.02

SM

.04

SM

.06Performance at Division of Research (task group 2)

T1 Time

FTP.01 and FTP.02 are done

on LAN (100Mbps)

Satellite TestsExpected Time - the previous results were scaled by a factor equal to the ratio of the Tachyon downlink QoS (384kbps or 1.544Mbps) to that observed in the T1 tests. For example, the expected time for SM.07 on 384/384kbps is 1s (Observed T1 time) x 1544kbps (T1 QoS) / 384kbps (satellite QoS) = 4sThe response time (or the latency) depends on three factors:

Propagation Delay (both satellite and Internet) – the time it takes for a packet to travel from one end to the other

Transmission Delay – the time it takes to send a packet onto the network. This depends on the bandwidth. For example: a 200 Bytes packet takes 200 (Bytes) x 8 (bits/Byte) / 384 (kbps) = 4ms

Processing Delay – the time it takes for servers, clients, or routers to process the packets (could be from 1 to 1000 ms)

The expected time computed above only reflects the Transmission Delay, i.e., the bandwidth.

384/384 v.s. T1

0

5

10

15

20

25

30

35

40

45

50

Rat

io

O.01

O.03

O.05

O.07

E.01

E.03

E.05

SM

.01

SM

.03

SM

.05

SM

.07

SM

.09

SM

.11

SM

.13

Task Number

Performance of 384kbps/384kbps compared to T1

O.01 O.02 O.03 O.04 O.05 O.06 O.07 O.08 E.01 E.02 E.03 E.04 E.05 E.06

SM.01 SM.02 SM.03 SM.04 SM.05 SM.06 SM.07 SM.08 SM.09 SM.10 SM.11 SM.12 SM.13 SM.14

Expected 384 time / T1 time ~ 4

Ratio = 384 time / T1 time

384/384 ALOHA v.s. T1

0

5

10

15

20

25

30

35

40

45

50

Ratio

O.0

1

O.0

3

O.0

5

O.0

7

E.0

1

E.0

3

E.0

5

SM

.01

SM

.03

SM

.05

SM

.07

SM

.09

SM

.11

SM

.13

Task Number

Performance of 384kbps/384kbps compared to T1

First column = 384 Second time / T1

time column = 384ALOHA

time / T1 time

Expected 384 time / T1 time ~ 4

1544/512 with ALOHA Task group I clock time

0

20

40

60

80

100

120

140

160

Clo

ck t

ime

(sec

onds

)

O.01

O.02

O.05

O.06

O.07

O.08

E.01

E.02

SM

.01

SM

.02

SM

.03

SM

.05

SM

.07

SM

.08

SM

.09

SM

.10

SM

.11

SM

.12

SM

.13

SM

.14Performance for 1544kbps/512kbps /w ALOHA (task group 1)

384/384 Observed time 384/384 w/ALOHA Observed time 1544/384 w/ALOHA Observed time

1544/512 with ALOHA Task group II clock time

0

5000

10000

15000

20000

25000

30000

Clo

ck t

ime

(sec

onds

)

O.03

O.04

E.03

E.04

E.05

E.06

FT

P.01

FT

P.02

SM

.04

SM

.06

Performance for 1544kbps/512kbps /w ALOHA (task group 2)

384/384 Observed time 384/384 w/ALOHA Observed time 1544/384 w/ALOHA Observed time

ThroughputIn 384kbps/384kbps with ALOHA, throughput of FTP and Network Drive approach 384kbps. On the other hand, the average throughput of SiteManager tasks is only 7kbps. Why?

0

50

100

150

200

250

300

350

400

Thr

ough

put(

kbps

)

O.01

O.03

O.05

O.07

E.01

E.03

E.05

FT

P.01

SM

.01

SM

.03

SM

.05

SM

.07

SM

.09

SM

.11

SM

.13

384kbps/384kbps /w ALOHA Throughput

SM.07 Open DWR on T1SM.07 – Open a blank Daily Work Report

SM.07 on T1: Efficiency is 16%, i.e., approximately 84% of the time the client is sitting there waiting

0 50 100 150 200 250 300 350 ms

0 20 40 60 80 100 120 140 160 ms

0 – 170 ms enlarged

Blue (outgoing traffic)Red (incoming traffic)

There are 12 request and response pairs sequentially

Propagation + server processing

delay ~ 8ms

Client processing

delay ~ 9ms

Propagation + server processing

delay ~ 20ms

Propagation delay

(roundtrip) ~ 7ms

SM.07 Open DWR on 384/384 ALOHA

SM.07 on 384/384 ALOHA: Efficiency is 2%, i.e., 98% of the time the client is sitting there waiting

0 2000 4000 6000 8000 10000 12000 ms

4200 4400 4600 4800 5000 5200 5400 5600 ms

4000-5600ms enlarged

Propagation + server processing delay ~

622 ms

Blue (outgoing traffic)Red (incoming traffic)

Packet longer than 200 bytes and has to reserve

before transmission

Packet shorter than 200 bytes and gets through via

ALOHA

Response Time Calculation/PredictionSuppose we know (in the case of SM.07) D - the total data to be sent and received (10kB) N1 - the number of request/response pairs with request smaller

than 200 Bytes (i.e, can be sent by ALOHA) (9) N2 – the number of request/response pairs with request larger

than 200 Bytes (3) N – the total number of request/response pairs (12) QoS - the Bandwidth provided (T1=1.5Mbps, S=384kbps) Tpropagation - the prop delay (7ms on T1, 1180ms on Sat) TpropagationALOHA - the propagation delay for ALOHA packets

(640ms) Ttotal_processing – the total processing time spent on client and

server (~200ms)

Response Time Calculation for SM.07

Then we can predict the response time to be Ttotal = Ttotal_propagation + Ttotal_transmission + Ttotal_processing

= (N x Tpropagation)+ D/QoS + 200= 12 x 7 + 10kB / 1.544Mbps + 200 = 84 + 52 + 200= 336 ms on T1

and= (N1 x TpropagationALOHA + N2 x Tpropagation) +

D/QoS + 200 = (9 x 640 + 3 x 1180) + 10kB / 384kbps +

200= 9300 + 208 + 200= 9708 ms on 384/384 ALOHA

Comparing these two we know the

determining factor is the total

propagation delay

0 2000 4000 6000 8000 10000 12000 ms

Actual observed response time

Response Time Calculation for SM.07

Using the above formula, we can compute the approximate response time for SM.07 under different QoS:

An increase of QoS from 100kbps to 1Mbps

corresponds to a decrease of response time from 10.9sec to

10.2sec (6%)

9.7 sec

Acceleration of SM.07

Note that we can’t change Tpropagation. In order to reduce the total propagation delay Ttotal_propagation = (N x Tpropagation), we can only reduce N.

If we can group these requests together (it should be possible since SM.07 doesn’t require any user input) and sent them all at once, i.e., make N=1, the response time would drop from 10 seconds to 2 seconds:

FTP.02 UploadFTP is a good example of sending multiple packets all at once, before waiting for acknowledgementsFTP02 – upload a 100MB file on 384/384 ALOHA (below shows a portion of the whole FTP session). The efficiency is 74%.

0-450ms enlarged

These ACKs are actually generated by IDU

Blue (outgoing traffic)Red (incoming traffic)

Waiting for ACK, local IDU buffer may fill up

SM.07 v.s. FTP.02SM.07

FTP.020 2000 4000 6000 8000 10000 12000 ms

Options to explore

Work with SiteManager developers to decrease “chattiness”: queries possibly could be aggregated and processed in parallel, instead of sequentially.Tachyon has reported that other users have observed improved performance with Oracle if the client does not query the database to check attributes of field data after editing each field of the form, but instead bunches those queries after form is filled out.

Dial (~53k) up vs. Tachyon

0

50

100

150

200

250

300

350

400

Clo

ck t

ime (

seconds)

O.0

1

O.0

2

O.0

5

O.0

6

O.0

7

O.0

8

E.0

1

E.0

2

SM

.01

SM

.02

SM

.03

SM

.05

SM

.07

SM

.08

SM

.09

SM

.10

SM

.11

SM

.12

SM

.13

SM

.14

384/384 ALOHA v.s. GlobalNet

T1 Time 384/384 w/ALOHA Observed time GlobalNet Time

Silverback Satellite Test

020406080

100120140

O.0

1

O.0

2

E.0

1

E.0

2

SM

.01

SM

.07

SM

.08

SM

.09

SM

.10

SM

.14

384/384 v.s. Silverback 1M/256 v.s. 1544/384

384/384 w/ALOHA Observed time Silverback 1544/384 w/ALOHA Observed time

0

200

400

600

800

E.0

3

E.0

4

SM

.06

384/384 v.s. Silverback 1M/256 v.s. 1544/384

384/384 w/ALOHA Observed time Silverback 1544/384 w/ALOHA Observed time

~27% of improvement on

SiteManager application

Not as good as Tachyon for

Windows Network Drive

Application

SM.07 on Tachyon 384ALOHA and Silverback

0 2000 4000 6000 8000 10000 12000 ms

There are three packets that can’t be sent via ALOHA,

which accounts for an additional

1.5 second

Tachyon 384/384 ALOHA

Silverback 1.5M/256

Inter-departure time for Tachyon and Silverback

Roundtrip time for Tachyon ranges from 600ms to 1400msRoundtrip time for Silverback ranges from 500ms to 700ms. This is due to the fact that

Silverback uses Deterministic TDMA for uplink and hence no reservation or contention is required. The idea is that each remote office is permanently assigned a time slot (with bandwidth generally less than 8Kbps). Additional bandwidth is then dynamically allocated by the hub.

Silverback hub is closer to Indianapolis Silverback uses a different kind of tunnel (it has encryption, but not standard VPN) than

Tachyon, which may reduces some processing time at gateways

Tachyon 384/384 ALOHA

Silverback 1.5M/256

SiteManager on Citrix v.s. on T1

0

5

10

15

20

25

30

Clo

ck t

ime

(sec

onds

)

SM

.01

SM

.02

SM

.03

SM

.05

SM

.06

SM

.07

SM

.08

SM

.09

SM

.10

SM

.11

SM

.12

SM

.13

SM

.14

Task number

T1 v.s. Citrix (1M/256k) for SiteManager

T1 Time Citrix

0

200

400

600

800

1000

1200

1400

1600

SM

.04

Since the actual data transmission occurs locally

at INDOT Headquarter (Fast Ethernet 100Mbps), the response time is even

shorter than running on T1

Keystroke delay

One-way Delay

Internet

Internet

Remote Officerunning terminal emulation (Citrix)

Hub Citrix Server

135 ms 135 ms ~50 ms

One issue with Citrix is the slow keyboard response, as can be seen from below: it takes half a second to see the input characters to show on the screen.

Silverback v.s. Tachyon on non-Citrix tests

Silverback performs worse than Tachyon on other tests, probably due to the following reasons:

Silverback does not completely terminate the local TCP session (as Tachyon does) by sending the local acknowledgement, hence does not accelerate TCP connection effectively

Silverback implements an end-to-end encryption, which reduces the bandwidth experienced by the user

0

20

40

60

80

100

120

140

Clo

ck t

ime

(sec

onds

)

O.01

O.02

O.05

O.06

O.07

O.08

E.01

E.02

Task number

384/384 v.s. Silverback 1M/256 v.s. 1.5M/512

384/384 w/ALOHA Silverback 1M/256k 1544/512 w/ALOHA

0

1000

2000

3000

4000

5000

6000

7000

Clo

ck t

ime

(sec

onds

)

E.03

E.04

FT

P.01

FT

P.02

Task number

Conclusion

Propagation delay is significant with satellite communicationSiteManager is very ChattyChatty * Propagation delaysluggish response, Regardless of bandwidthIn contrast an application like FTP works quite well in high latency environmentsConsideration should be given to reducing the amount of IP chatter generated by SiteManager to improve Satellite performance.Alternatively, terminal service applications (i.e. Citrix) can be used as a work around.

Propagation Delay

Request for time slot

Actual data transmission

540 ms

+ 640 ms

1180 ms !!

One-way Delay

Internet

Internet

Remote Office(West Lafayette)

Tachyon(San Diego)

INDOT HQ(Indianapolis)

135 ms 135 ms ~50 ms

ThroughputIn 384kbps/384kbps with ALOHA, throughput of FTP and Network Drive approach 384kbps. On the other hand, the average throughput of SiteManager tasks is only 7kbps. Why?

0

50

100

150

200

250

300

350

400

Thr

ough

put(

kbps

)

O.01

O.03

O.05

O.07

E.01

E.03

E.05

FT

P.01

SM

.01

SM

.03

SM

.05

SM

.07

SM

.09

SM

.11

SM

.13

384kbps/384kbps /w ALOHA Throughput

Questions

Darcy Bullock, P.E.Professordarcy@purdue.edu765 494 2226

Appendix B - Error BarThe error bar shows 95% confidence interval

384/384 Observed time and error bar (task group 1)

0

20

40

60

80

100

120

140

160

O.01

O.02

O.05

O.06

O.07

O.08

E.01

E.02

SM

.01

SM

.02

SM

.03

SM

.05

SM

.07

SM

.08

SM

.09

SM

.10

SM

.11

SM

.12

SM

.13

SM

.14

Clo

ck t

ime

384/384 Observed time

Appendix B - Error Bar Cont’d

384/384 Observed time and error bar(task group 2)

0

500

1000

1500

2000

2500

3000

O.0

3

O.0

4

E.0

3

E.0

4

FT

P.0

1

FT

P.0

2

SM

.06

Clo

ck t

ime

384/384 Observed time

384/384 Observed time and errorbar (task group 3)

0

5000

10000

15000

20000

25000

30000

35000

E.0

5

E.0

6

SM

.04

Clo

ck t

ime

384/384 Observed time

Appendix C - Inter-departure and inter-arrival time

SiteManager Inter-departure and inter-arrival time distribution on 384/384 ALOHA

ACKs send by IDU

Packets sent via ALOHA

Packets that can’t be sent via ALOHA

Test Tasks

Appendix A – Task Description

O.01 Incoming mail with a 900KB attachment (indotrs4\smgr\Training_Photos_030407\DSCN0465.jpg)O.02 Outgoing mail with a 900KB attachment (indotrs4\smgr\Training_Photos_030407\DSCN0465.jpg)O.03 Incoming mail with a 7.66MB attachment (viter.pdf)O.04 Outgoing mail with a 7.66MB attachment (viter.pdf)O.05 Calendar / Jan 20 / double click 12 am (clock starts now)O.06 Calendar / Jan 20 / double click 12 am / Subject: test, Content: paste the first two pages from tachyon_testplan_jan04_v4.doc / Close and Save (clock starts now)O.07 Public Folders / All Public Folders / Agency Information / Vendors Contact List (clock starts now)O.08 Public Folders / All Public Folders / Agency Information / Vendors Contact List -> Double click Burns & McDonnell (clock starts now)E.01 Copy \\indotrs4\smgr\Training_Photos_030407\DSCN0465.JPG (900KB) and paste it onto desktopE.02 Copy and paste DSCN0465.JPG (900KB) to \\indotrs4\smgr\BWTestE.03 Copy \\indotrs4\smgr/smdbl00.db (6608 KB) and paste it onto desktopE.04 Copy and paste smdbl00.db (6608 KB) to \\indotrs4\smgr\BWTestE.05 Copy TESTFILE.zip (763330 KB) from \\indotrs4\smgr\BWTest and paste it onto desktopE.06 Copy and paste TESTFILE.zip (763330 KB) to \\indotrs4\smgr\BWTestFTP.01 Download bigfile.dat (100000 KB)FTP.02 Upload bigfile.dat (100000 KB)

Appendix A – Task Description Cont’d

SM.01 Run SiteManager / Server Mode / User name and password (clock starts now)SM.02 Accessories / In-Box / Service / Compose / To: Chen, T.C. / Subject: Test / Message: copy and paste the first two pages from tachyon_testplan_jan04_v4.doc / Services / SendSM.03 Accessories / In-Box / double click the mail sent in procedure SM.02 (clock starts now)SM.04 Pipeline and Zip / server to PM pipeline / open Contract R-26013-G / Check Basic

Contract Data -> Service -> PipelineData (clock starts now)SM.05 Pipeline and Zip / PM to Server Pipeline / open Contract R-26013-G / Check Basic

Contract Data / Service / PipelineData (clock starts now)SM.06 Change orders / Change order Maintenance / Change Order Item / New Contract Item tab / Right click item code to do a search (clock starts now)SM.07 Daily Work Report / Daily Work Report (clock starts now)SM.08 Daily Work Report / Daily Work Report / Change from Info view to Contractor view SM.09 Daily Work Report / Daily Work Report / Change from Daily Staff view to Work Items view SM.10 Daily Work Report / Daily Work Report / Double click Construction Engineering under Work Items view (clock starts now)SM.11 Daily Work Reports / Diary / Check Authorized / Save (clock starts now)SM.12 Contractor Payments / Estimate / Generate Estimate / Services / Generate EstimateSM.13 Contractor Payments / Process List / Select R-26013-G and double click to submit (clock starts now) / Process Submitted click ok / Process Completed (clock stops now)SM.14 Contractor Payments / Reports / Summary by Project / Double Click R-26013-G / Select Estimate Nbr 0001 (clock starts now)

EXAMPLE Task Duration via Satellite

384/384 Results (task group 1)

0

20

40

60

80

100

120

140

160

Clo

ck t

ime

(sec

onds

)

O.01

O.02

O.05

O.06

O.07

O.08

E.01

E.02

SM

.01

SM

.02

SM

.03

SM

.05

SM

.07

SM

.08

SM

.09

SM

.10

SM

.11

SM

.12

SM

.13

SM

.14

Performance for 384kbps/384kbps (task group 1)

T1 Time Expected time 384/384 Observed timeThe main time contributor of

SM.12 and SM.13 is the server (generate estimate and report),

hence they are not seriously affected by bandwidth and

propagation delay.

384/384 Results (task group 2)

0

5000

10000

15000

20000

25000

30000

Clo

ck t

ime

(sec

onds

)

O.03

O.04

E.03

E.04

E.05

E.06

FT

P.01

FT

P.02

SM

.04

SM

.06

Performance for 384kbps/384kbps (task group 2)

T1 Time Expected time 384/384 Observed time