MobiBed: An Open Network Protocol Testbed For Mobile Phones In TheWild

Andong Zhan, Marcus Chang, Da Zheng, Andreas TerzisDepartment of Computer Science

Johns Hopkins UniversityBaltimore, MD 21218

{andong, mchang, zhengda, terzis}@cs.jhu.edu

AbstractSmartphones are becoming the dominant devices peo-

ple use to access the Internet. Following this trend, adeep investigation of network protocol performance oncellular networks is becoming increasingly important.However, most of the current protocol testing on smart-phones are based on laboratory scope testbeds whichrequire root privileges or even customized firmwares.These constraints limit such tests within a laboratoryscope. Unfortunately, small scale testing cannot provideresults with high confidence due to the high spatial andtemporal variability of cellular networks.

To address this challenge of testing network protocolsat scale, we design and implement MobiBed, an opennetwork protocol testbed for mobile phones in the wild.Each phone participating in MobiBed runs a networkprotocol emulator in user space that implements sand-boxed network stacks. Protocol packets are encapsulatedin UDP tunnels that require no kernel changes or super-user privileges. The paper evaluates the performance ofMobiBed’s protocol testing environment and shows thata MobiBed TCP implementation is comparable with thekernel based equivalent up to link speeds of 10 Mbit/sec,and shows only a 3% overhead at speeds up to 4 Mbit/s.Through several examples we also show how MobiBedcan facilitate network protocol debugging and develop-ment.

1 Introduction

Smartphones are quickly becoming the primary waypeople access the Internet. According to a recent sur-vey [28], 88% of U.S. adults own a cell phone of somekind as of April 2012, and more than half of these cellowners (55%) use their phone to go online. And 31% ofthese users say that they mostly go online using their cellphone, and not using some other device such as a desktopor laptop computer. Another mobile usage statistics [2]

also show 75% of all emails, 60% of all Facebook posts,and 90% tweets are authored from mobile devices. Thisshift in user behavior coupled with the fact that cellu-lar networks are very different than wired and even WiFinetworks, means that understanding the performance ofexisting and future network protocols in the wild is criti-cal.

Unfortunately, it is not straightforward for researchersto understand and evaluate the performance of networkprotocols on cellular networks due to several reasons.First of all, carrier networks use different technologies(e.g., 1xRTT, EVDO, GPRS, EDGE, HSPA, LTE, etc.).Moreover, carriers configure their networks differently,optimizing for different sets of constraints. The resultis that protocols that perform well in one network maysuffer in others. Furthermore, even when accessing thesame network, link quality varies over time and space asa function of radio signal quality and network use. Theresults of our measurement study in Section 3.1, illus-trate this variability across carriers, space, and time. Thisadvocates building a testbed that can be scaled to largegeographic areas and not just limited to a stationary lab-oratory.

The challenge in building such a testbed is providingall the functionalities available in lab-scale testbeds, butin an unsupervised and non-privileged public environ-ment such as third-party smartphones. In current lab-scale testbeds, researchers can fully control their smart-phones by rooting the operating systems, or even in-stalling custom firmwares. Although these approachesare cumbersome, they enable researchers to develop andfine tune network protocols inside the kernel and testthem in a controlled environment. However, these ap-proaches do not scale to large numbers of smartphones,especially for the phones in the wild owned by third par-ties. Due to these limitations, current open and scalabletools, e.g., MobiPerf [19] and iPerf for Android [13] onlyprovide simple network measurement tools like Ping andTCP throughput tests.

1

User Space

Kernel Space

Application

TCP/UDP

Current MobiBed

Touchable Untouchable

TCP/UDP

IP

Application

IP

MobiBed

Driver

UDP

IP

Driver

Figure 1: MobiBed utilizes the UDP packet interface toemulate a raw socket. By moving the network stack touser space, protocol evaluation and development can beperformed without root privileges.

In this paper, we present MobiBed, a testbed for de-veloping and evaluating network protocols in the wild,which relies on crowdsourcing to overcome scalabilityproblems. As Figure 1 shows, the basic idea behind howMobiBed can provide access to the raw packet interfacewithout requiring root privileges is to use standard UDPpackets to emulate the underlying raw packets only ac-cessible through the kernel. By building network proto-cols in user space (instead of in kernel space) it becomeseasier and more secure for researchers to modify, replaceand monitor these protocols. This design also makes itpossible for researchers to remotely distribute their cus-tomized protocols into the phones of people that alreadyhave the MobiBed client installed.

To realize this idea, we modify the J-Sim [30] simula-tor to become the underlying protocol testing frameworkby porting it to Android and extending it with access tothe real network and not just a simulated one. ThroughJ-Sim, researchers can build their custom network proto-cols in Java, first debug them locally using the simulatorand subsequently upload them to the MobiBed testbedand have them evaluated in the wild.

We validate MobiBed’s performance in practice bycomparing a user-level TCP implementation to the TCPin Android’s Linux kernel and show that its perfor-mance is on par for typical cellular network bandwidths.From our experience in testing different TCP congestioncontrol variants and re-implementing the Dynamic Re-ceive Window Adjustment (DRWA) algorithm proposedin [23], we demonstrate through several examples howMobiBed can be an effective testing and debugging toolfor protocol modification and development.

The main contributions of this paper are: (1) Wepresent MobiBed as the first viable mobile protocoltestbed in the wild which provides a complete network

protocol testing solution for cellular networks, includ-ing a general mobile network protocol distribution plat-form and testing environment; (2) We validate the per-formance of MobiBed in practice and show how it canimprove mobile protocol testing.

This paper has five more sections. In Section 2 wepresent related work in this area. Section 3 and Section 4discuss MobiBed’s overall design and actual implemen-tation details, which we evaluate the performance of inSection 5. Finally we conclude the paper in Section 6.

2 Related Work

Wireless network protocol testbeds have emerged as alogical extension to the comprehensive testbed infras-tructure already in place for wired networks, whereboth global (PlanetLab [6, 9] and GENI1) and local(DOME [29]) testbeds have been extended with wirelessnodes in the form of 3G and WIMAX connections. Al-though the connections in these testbeds became wirelessthe nodes, however, remained PC class devices.

This limitation was removed in SmartLab [8] whichallows users to remotely instrument a testbed consistingof 40 Android smartphones. In MUSA [14] the authorswent a step further and included their own 3G cellularbasestation as part of the testbed thereby enabling themto monitor both ends of the wireless connection.

As much as a testbed environment provides a con-trolled laboratory for network protocol testing the staticenvironment lacks the dynamic spatio-temporal perfor-mance variations observed in practice. On the otherhand, as a crowdsourced testbed, MobiBed’s nodes areexposed to the exact environmental changes that networkprotocols are exposed to in real life.

Other smartphone frameworks, such as AnonySen-se [10], MobiPerf [20, 21], Medusa [25], and CITA [26]also rely on crowdsourcing. However, these systemsonly allow scripting of functions that have already beendistributed with the original application where MobiBedon the other hand leverages the full expressibility of theJ-Sim simulator and allows Java classes to be run withoutupdating the original application.

This fidelity is similar to what is provided by bothCrowdLab [11] and PRISM [12], where the former uti-lizes a XEN hypervisor to virtualize the access to NokiaN810 tablets while the latter allows execution of arbi-trary binary code on Android devices. However, in or-der to enable this flexibility all the devices have to beequipped with either custom firmware or be jailbroken,which limits the feasibility for widespread use in gen-eral and crowdsourcing in particular. This distribution

1Global Environment for Network Innovations, http://www.geni.net

2

Phone Mean (ms) Std (ms) PRR (%)home/AT&T UMTS 559.39 482.27 98.57home/Verizon EVDO 439.89 1918.2 97.37home/wired 18.38 15.47 97.59lab/wired 7.00 9.95 99.19lab/Verizon EVDO 104.19 66.72 98.37lab/Verizon LTE 118.94 158.63 99.1

Table 1: Mean, standard deviation, and packet receptionratio (PRR) from the RTT measurements.

problem is the same one faced by Falaki et al. [17] andLiveLab [27] because both require root privileges.

On the other hand, since MobiBed is specifically de-signed for network protocol testing and utilizes UDP tun-neling to emulate the raw socket interface that requiresroot privileges, MobiBed can be distributed and used asa regularly user application without the need for any spe-cial installation procedures.

3 System Design

First, we present some preliminary results of the cellu-lar network variety across multiple providers in order toshow the necessity for large scale protocol testing and toguide the design of MobiBed. Next, we present a high-level description of the overall architecture and programflow. Last, we present the actual sandbox in which thenetwork protocol emulation is performed and show howto compose protocol testing tasks by combining networkprotocol modules.

3.1 Cellular Network MeasurementsIn order to illustrate the cellular network variation, weran several experiments on a wide range of smartphones(Motorola Droid, Samsung Nexus S, Samsung NexusGalaxy, and HTC ONE X) and cellular networks (AT&T3G, Verizon 3G, and Verizon 4G LTE) in the UnitedStates.

First, we measured the latency fluctuations during aweek from our lab and one of the author’s home to vari-ous websites (google.com, youtube.com, facebook.com,and our group’s web server). Using four phones con-nected to different cellular networks and two PCs con-nected to the wired network we measured the round-trip-time (RTT) every five seconds during that week. Next,we ran a week-long TCP throughput test using the samesetup to repeatedly download a 1 MiB file from ourgroup’s web server every 20 min.

The results for the first experiment are shown in Ta-ble 1 and Figure 2 and the results for the second in Fig-ure 3. From these results we make the following obser-vations:

Cloud (Server)

Mobile Devices

MobiBed Script MobiBed Code

MobiBed Server

MobiBed Client

Tasks and

Results

MobiBedTask

MobiBed Sandbox

Checkin

MobiBedTask

Internet

Figure 4: MobiBed System Architecture.

Cellular networks are much more variable thanwired networks. From Table 1, we see that althoughthe packet reception ratio (PRR) on both the cellular net-works and wired networks are similar, the mean and stan-dard deviation of the latencies from the cellular networksare 1-2 orders of magnitude higher than the ones from thewired networks.Network quality varies across different networks andlocations. Not surprisingly, the network throughput fromdifferent network types (e.g., AT&T UMTS, VerizonEVDO, and Verizon 4G LTE) are significantly differ-ent as shown in Figure 3(a). However, Figure 2(a) alsoshows that the location has a significant influence onthe wireless network quality while the wired network re-mains unaffected.Network quality in cellular networks shows a hightemporal variation. Next we group the latency andthroughput by hour of the day (for multiple days) to showthe temporal correlations, with the largest correlationsfound in the latencies as seen in Figure 2(b) which alsoclearly shows that latencies are highly correlated to thetime of day. Specifically, the network experiences thehighest latencies from the midnight to 2:00 and the bestperformance between 12:00 and 14:00. For the through-put, as shown in Figure 3(b), the download performancefrom 4:00 to 6:00 is significantly higher than all the otherperiods.Based on these observations, a comprehensive protocoltesting environment on cellular networks should be ableto cover different network types, scale to a large geo-graphic area, and be able to operate for extended periodsof time. While a lab-scale testbed can satisfy the firstand last requirements it lacks the geographic coverage tosatisfy the second requirement.

3

0 100 200 300 400 5000

0.2

0.4

0.6

0.8

1

RTT (ms)

CD

F

home/ATT UMTS

home/Verizon EVDO

home/wired

lab/wired

lab/Verizon EVDO

lab/Verizon LTE

(a) The CDF of the RTTs collected from AT&T and Verizon net-works at an author’s home and our lab. The wired links show morestable RTTs while the RTTs from the cellular networks are signifi-cantly larger and show higher variance across different networks andlocations.

0 100 200 300 400 5000

0.2

0.4

0.6

0.8

1

RTT (ms)

CD

F

0−2

4−6

12−14

20−22

(b) The CDF of the RTTs grouped by time (two hour bins) from thehome for Verizon’s EVDO over a week. Cellular network latencyhas a high temporal variation. For example, RTTs collected between0:00 to 2:00 are much larger than the ones collected between 12:00to 14:00.

Figure 2: RTT measurements collected by pinging google.com every five seconds over a week.

0 500 1000 1500 2000 2500 3000 35000

0.2

0.4

0.6

0.8

1

Throughput (kbps)

CD

F

lab/Verizon LTE

lab/Verizon EVDO

lab/AT&T UMTShome 1/Verizon EVDO

home 2/Verizon EVDO

home 2/Verizon LTE

(a) The CDF of TCP throughput on different networks and loca-tions.

0 500 1000 1500 20000

0.2

0.4

0.6

0.8

1

Throughput (kbps)

CD

F

0−2

4−6

12−14

20−22

(b) The CDF of TCP throughput grouped by time (two hour bins)from the lab/Verizon EVDO dataset. The layered curves show astrong temporal variation.

Figure 3: Downlink throughput measurements collected by downloading a 1 MiB file from our department websiteevery 20 minutes over a week.

3.2 MobiBed Architecture

In order to enable large scale network protocol testing,with regards to both geographical area and participat-ing nodes, MobiBed is structured around a collection ofservices running on both the cloud and on smartphones,whereby a client application on each participating smart-phone is connected to a central server in the cloud.

Figure 4 shows the steps taken to perform a new ex-periment. First, a new task is created by submitting itscontents to the MobiBed server through the web GUI.The contents of a task consist of two parts: a script anda packaged binary code file. The script is used to de-fine, configure, and link the modules needed by the task.The binary code file is optional and only used to delivercustom modules to MobiBed. Besides the task’s content,the user also defines policies for each test, including theschedule (when to execute the script), and network type(specifying the network type and phone, e.g., only exe-cute in 4G LTE networks). Once the task has been sub-mitted to the server, it is saved in the database and is

ready to be distributed to the smartphone clients.

Submitted tasks then are distributed to the MobiBedclients running on participating smartphones. This clientapplication can be downloaded from the app store andruns as a regular user space application without need forspecial user privileges. The MobiBed client will run asa background service and synchronize with the MobiBedserver periodically. We call this process check-in. Duringthe check-in process, the client updates its current phoneand network properties as well as the tasks needed to runon the device.

Based on the schedule, the MobiBed client launchesnewly added tasks and generates a new sandbox environ-ment for each task, responsible for loading the any bi-nary code and interpreting the script. When the task hasbeen executed, the collected results are stored locally anduploaded to the central MobiBed server during the nextcheck-in event. MobiBed also enables phone owners toblock uploading results unless the phone is charging andconnected to WiFi.

4

Bulk Sink

TCP Client

MoBiBed Socket

To Internet

down@

up@

up@

down@

MSS = 512B

SACK = true recv@

pcap@

(a) The Design of TCP Client.

Bulk Source

TCP Server

MoBiBedSocket

To Internet

down@

up@

up@

down@

MSS = 512B

SACK = true cwnd@

pcap@

DataSize = 5MB

Component

Data port

Event port

File Writer

(b) The Design of TCP Server.

Figure 5: An example of a MobiBed Task: Bulk Data Transmission using TCP.

3.3 Protocol Sandbox Environment

The actual sandbox where the network protocol emula-tion is performed is based on J-Sim, the Java implemen-tation of the open component-based software architec-ture, Autonomous Component Architecture (ACA) [30,31]. J-Sim has been used in network simulations andsoftware-programmable routers with well defined com-position semantics and running performance [31, 15].

We choose J-Sim for the following reasons. First, asan open network simulation tool, J-Sim already containsa set of built-in protocol modules that can be used imme-diately in protocol testing. Second, by integrating J-Simwith MobiBed, a protocol can be first simulated in J-Simand then subsequently tested on MobiBed directly with-out any changes. Furthermore, because of Java’s plat-form independence, the MobiBed sandbox can be de-ployed on not only smartphones but also other platforms,such as PCs and servers, meaning new network protocolscan be tested immediately on servers and as well as com-patible software based network components. This is themain reason we chose not to use other well known en-vironments such as Click [24]. Finally, J-Sim alreadyprovides a script interface allowing integration with dif-ferent script languages, e.g., we use the Tcl scripting lan-guage to combine components in runtime. This is idealfor MobiBed because we can tune or modify protocolsbeing tested by changing the script without having to re-install the sandbox.

The basic J-Sim entity is a component and each com-ponent owns one or more end points, called ports2. Asoftware system is composed of a collection of com-ponents communicating with each other via their ports,analogous to an IC chip design. When data arrives at a

2port id is indicated as id@groupid, and id@ means the port iswithin the default group.

port, the port’s host component processes the data im-mediately in an independent execution context. Theseprocesses can run simultaneously and their interferenceis only subjected to synchronization and mutual exclu-sion. The component is autonomous because it can han-dle data in independent execution contexts with all com-ponents bound to each other only at the system configu-ration time. In addition, J-Sim also supports compositecomponent, which means one component can consist ofseveral smaller components and the entire system formsa component hierarchy.

In order to transform J-Sim from a purely network pro-tocol simulation environment into an emulation environ-ment, we created a socket component with access to thephysical network, called MobiBedSocket, which utilizesthe standard user-space UDP communications to emulatekernel raw sockets. This basic network interface can thenbe used by all other modules in J-Sim.

MobiBedSocket triggers a thread listening on a par-ticular network port for incoming UDP packets and sub-sequently decapsulates them into Java packet objects; atthe same time, MobiBedSocket also encapsulates pend-ing packets from other modules into UDP packets andsend them over the Internet.

We note that Dong et al. [15] introduced a similarsocket component in J-Sim called JSocket, which alsobridges the simulated network with the real one. How-ever, since JSocket uses the native Java raw socket tobridge the two networks root privileges are required,which make it unsuitable for our needs.

Figure 5 shows an example of a MobiBed task: TCPbulk data transmission. In Figure 5(a), the applicationprotocol BulkSink and the transportation protocol TCP-Client are two modules3. We connect the down@ port

3According to J-Sim, modules are components with two default

5

of BulkSink with the up@ port of TCPClient at the con-figuration time, i.e., in the script (Line 16 in Figure 6),so that the received data from TCPClient can be for-warded to the application layer. In the MobiBed sand-box, users can easily interchange different network pro-tocols. For example, users can change TCP to UDP orchange BulkSink to FTP. Additionally, users can instru-ment event ports to export important events for offlineanalysis. For example, in Figure 5(b), we want to knowhow the congestion window size changes, so we attachthe in@ port on a FileWriter component with the cwnd@port in order to write the event log to a file. To help usersdiagnose their protocol while running, we also created apcap@ port in MobiBedSocket to export all packets sentin the pcap format, similar to what tcpdump [5] can do inthe kernel.

An important point we would like to emphasize is thatsmartphone owners always have higher priority. In Mo-biBed, MobiBedSocket is the only interface connectedwith the real network. The sandbox restricts any othercomponents from calling any of the JVM network API,and user defined components are also statically verifiedbefore loaded into the sandbox. In addition, in order toensure that the testing task is not obstructing the phone’sdaily usage while running, we enforce severe restrictionson the sandbox. During the interpretation and execution,the sandbox will automatically shutdown when one ofthese three conditions are met: (1) an error is generated,(e.g., TCP connection error); (2) resource usage exceedsthe quota (e.g., passing the data usage limit); (3) the exe-cution time exceeds a predefined limit.

4 Prototype Implementation

In this section, we describe our MobiBed prototype im-plementation in details. We implemented the MobiBedserver as a Web app using the Google App Engine frame-work and the mobile client on the Android platform.

4.1 Task Distribution and Result Collec-tion

We implement the MobiBed task distribution and resultcollection as an extension of MobiPerf 2.0 [3], which isalso an open source application for measuring networkperformance on mobile platforms. Currently, MobiPerfonly provides simple network measurements tools suchas ping, traceroute, and throughput tests, and does notsupport custom network protocols or any other user sup-plied code, however, MobiPerf does support a task dis-tribution and result collection similar to what we seek.

ports: up port and down port

1 # tcp_sender.tcl

2 # Create protocol modules

3 mkdir drcl.inet.application.BulkSource app

4 mkdir drcl.inet.transport.TCP tcp

5 mkdir drcl.inet.mobibed.MobiBedSocket sock

6 mkdir drcl.inet.tool.PCapTrace pcap

7 mkdir drcl.comp.tool.FileWriter fw

89 # configure modules

10 ! tcp setPeer "xxx.xxx.xxx.xxx" 9418

11 ! tcp setImplementation "CUBIC"

12 ! tcp setMSS 512

13 ! tcp setSackEnabled true

1415 # connect components

16 connect -c app/down@ -and tcp/up@

17 connect -c tcp/down@ -and sock/up@

18 connect -c tcp/cwnd@ -to fw/0@1

1920 # Attaches pcap trace

21 attach pcap/in@ -to sock/pcap@

2223 # set debug

24 setflag debug true -at "dupack" tcp

2526 # Attaches runtime

27 attach_runtime .

28 run .

Figure 6: An example Tcl script of the TCP sender

In order to enable generic protocol testing on Mo-biPerf we modify the web GUI and database to supportuser defined tasks containing custom binary code and wealso modify the mobile client to download and setup upthe sandbox protocol testing environment.

4.2 Protocol Testing Environment

As we mentioned in Section 3.3, we have to build theMobiBedSocket module in order to bridge the gap be-tween the simulated network in J-Sim and the real net-work accessible through the JVM.

When a MobiBedSocket instance is created in J-Sim,the J-Sim runtime will call its start() function at the be-ginning of the test. Within the start() function, Mo-biBedSocket opens a new exclusive thread for a Data-gramSocket continuously receiving UDP packets on auser defined port number. When new data/packets ar-rive at the up port of the MobiBedSocket, it uses thisDatagramSocket to forward them to the Internet. In Mo-biBedSocket, we use two pre-allocated buffer arrays tocache sending and received packets respectively (sincespace allocation and garbage collection are expansive onAndroid Dalvik VM). So one MobiBedSocket can sup-port parallel requests from multiple protocol modules

6

during one test and fully utilize the DatagramSocket.Since all packets running in J-Sim are Java objects,

which cannot be directly transmitted over the Internet,we incorporate a packet format conversion function intoMobiBedSocket. This function converts packets betweenthe J-Sim formats and the standard network packet for-mats, including full header encapsulation, on top of theUDP format. For example, a TCP packet in J-Sim willconvert to a UDP packet payload containing both TCPand IP headers. Therefore, compared to the size of a nor-mal TCP packet, the MobiBed TCP has the additionaloverhead of the UDP and IP headers.

Although J-Sim is a Java based simulation tool, it can-not run directly in the Android Dalvik virtual machinebecause of several library conflicts, specifically, the Tclinterpreter in J-Sim is not compatible with the DalvikVM in Android.

In order to port J-Sim to Android, we had to replacethe conflicting libraries and built-in Tcl interpreter withone written completely in Java (JTcl [1]).

To optimize the transportation protocol testing in J-Sim, we made some major changes to the TCP mod-ules. First, we modified the main TCP class in J-Sim byusing a new architecture introduced in Linux 2.6.13 byStephen Hemminger called pluggable congestion mod-ule [4]. Previously the congestion algorithms in J-Sim,including Reno, New Reno, and Vegas, were all mixedtogether in one single Java source file. Our new Javabased pluggable congestion module facilitates rapid cus-tomization and interchangeability.

This also makes code transfer between J-Sim andLinux TCP easier because they both use the same codestructure. Using this pluggable congestion module, wealso implemented CUBIC [18], the TCP variant currentlyused in the Linux kernel. Since the current Androidare using CUBIC as the default congestion control algo-rithm, MobiBed users can play with CUBIC as same asin the Android kernel space. Besides, we also implementother TCP standards, e.g., the three way handshaking forTCP connection establishment and termination, and TCPoptions for Selective Acknowledgment (SACK), windowscaling, and timestamp for RTT measurement, etc.

Note that as an emulator based on UDP, protocols inthe MobiBed sandbox cannot directly connect to otherservers, e.g., connect to google.com via HTTP, but haveto go through a gateway, similar to VPN tunneling.

Finally, to increase usability and verify the customcode before submitting it to the MobiBed server, we pro-vide a tool to help users package their class files into a jarfile executable in Android. Basically, the tool is a shellscript using the dex2jar utility to convert Dalvik byte-code to Java, and the javap program from the Sun JDKfor static program analysis. Any class invoking Java orAndroid network APIs (e.g., textttjava.net.socket) will be

103

104

105

103

104

105

1955

3912

7815

19431

28649 3010938800

1951

3813

69918389 7840 7989

9321

Link Bandwidth (kbits/sec)

Th

rou

gh

pu

t (k

bits/s

ec)

Link Bandwidth vs Throughput

Kernel Throughput

MobiBed Throughput

Figure 7: The throughput of both the kernel stack andMobiBed are shown when the link bandwidth increases.MobiBed can achieve similar performance as the kernelwhen the bandwidth is less than 10 Mbit/sec.

blocked during the process.

5 Evaluation

We first evaluate the performance of the MobiBed proto-col testing environment and show the overhead. Then,we show through several examples how MobiBed canimprove the protocol testing on mobile phones.

5.1 PerformanceBecause MobiBed runs network protocols in user space,extra overhead is introduced to the system in the form ofmore data copying and longer packet headers. Not sur-prisingly, MobiBed’s network stack will be slower thanthe equivalent kernel stack, however, we will show thatfor throughput and latencies of the same order of magni-tude as those observed in our preliminary experiment de-scribed in Section 3.1 (i.e., 3 Mbit/s throughput and 100ms latency) this overhead is negligible and that MobiBedis capable of recovering protocol insights that previouslyrequired root access.

To quantify this overhead, we first setup a TCP bulkdata transmission test in a controlled environment using aThinkPad T510 laptop running Ubuntu 12.10 as the TCPsender and a HTC ONE X smartphone as the TCP re-ceiver, with the phone connected directly to the laptopvia a USB cable.

We compare the download throughput of the MobiBedTCP with the native Java TCP (which invokes the Linuxkernel TCP) at different bandwidths and show the resultsin Figure 7. In both tests we use a 3 MB receive buffer,enable SACK, use the CUBIC [18] congestion avoidancealgorithm, and Dummynet [7] to restrict the link band-width. We see that for bandwidths less than 8 Mbit/s thetwo network stacks have similar performance. Specifi-cally, MobiBed TCP is 3% less efficient at 4 Mbit/s and

7

10−1

100

101

102

0

10

20

30

40

50

60

70

80

Link Latency (ms)

Applic

ation L

ate

ncy (

ms)

Link Latency vs Application Latency

Kernel Latency

MobiBed Latency

Figure 8: The application latency of the kernel stack andMobiBed is shown. MobiBed is 1 ms slower than thekernel network stack on average, due to additional over-head.

11% at 8 Mbit/s. At higher bandwidths the MobiBedTCP levels out around 10 Mbit/s while the Java TCP firstlevels out at 40 Mbit/s. This discrepancy is probablydue to the unavoidable overhead from copying contentbetween packets and headers when tunneling MobiBedTCP through UDP packets. This is supported by Ely etal. [16] which also saw similar performance degradationin their user space network protocol, even though theyutilized root privilege to gain access to the raw socketinterface in the kernel. Since our preliminary measure-ments (shown in Figure 3(a)) suggest that the averagethroughput in the tested cellular networks are all less than3 Mbit/s our MobiBed testbed is indeed efficient enoughto support protocol testing under these conditions.

Figure 8 depicts how the extra data copies and headeroverhead affect latency. In this experiment, we use thesame setup as in the throughput test above, but we com-pare the latency differences instead. Since the origi-nal latency is almost zero through the USB cable, weuse Dummynet to vary the link latency from 0.2 to 64ms. The kernel latency is measured using the ping com-mand while in MobiBed we measure how long it takesto forward a packet through the MobiBedSocket runningon the laptop and then receive the same packet echoedback by the MobiBedSocket running on the phone. Mo-biBed’s latency is 1 ms larger than the kernel latency inaverage and the maximum difference is less than 1.5 ms.Therefore, the additional latency of MobiBed is also neg-ligible for protocols running on cellular networks.

5.2 Protocol Testing on Smartphones

With a customizable network stack running in user spaceMobiBed enables a wide range of protocol investigationpossibilities. Below we will show through several exam-ples exactly how insightful this tool can be.

0 20 40 60 80 100 1200

20

40

60

80

100

Time (Second)

Congestion W

indow

Siz

e (

KB

yte

s)

TCP_CubicTCP_NewRenoTCP_Vegas

Figure 9: Different TCP variants on MobiBed on the Ver-izon EVDO network (tcp rmem max = 110208). Thecongestion windows collected from MobiBed show thesame patterns as the ones from the kernel TCP in [23].

Packet Capture In Linux, superusers can use tcpdump

to capture packets in a .pcap file for offline analysis,e.g, view them in WireShark. To accomplish this on anAndroid phone, first the phone would have to be rooted,which is a tedious process that differs from phone tophone and Android versions and also voids the phone’swarranty. Next, tcpdump would have to be installed onthe phone.

Meanwhile, capturing packets in MobiBed is a muchsimpler process. After installing the MobiBed applica-tion from the app store, packet capturing can be per-formed by adding just two lines to the Tcl script as shownin Figure 6. Specifically, in Line 6 a new tool is createdcalled PCapTrace, that exports all the packets in a .pcapfile, and in Line 21 its in@ port is attached to MobiBed-Socket’s pcap@ port.TCP Congestion Window Monitoring In order to mon-itor the TCP congestion window (cwnd) in Linux onewould have to use the kernel module tcpprobe. On anAndroid phone, this would require a customized kernelwith tcpprobe enabled which can only be loaded by in-stalling a custom firmware. In contrast, to monitor cwndin MobiBed, we only need to create a FileWriter tool(Line 7 in Figure 6) and connect it to the cwnd@ port inthe TCP module (Line 18). Once the test is over, the logfile can be accessed through the MobiBed web server. Infact, the cwnd data shown in Figure 9 and Figure 10 iscollected this way.Protocol Debugging By using an actual network simu-lator as foundation for our sandbox, MobiBed also in-herits the user friendly debugging interface from J-Sim.Line 24 in Figure 6 sets the debug flag for the TCP mod-ule and the debug level to “dupack”, i.e., print out debugmessages related to duplicate ACKs. Since these debugmessages are printed to the standard output, on Android,logcat can be used for interactive debugging. Alterna-tively, this debug trace can be stored automatically to theSD card or collected through the MobiBed web interface

8

0 50 100 1500

200

400

600

800

1000

Time (Second)

Congestion W

indow

Siz

e (

KB

yte

s)

TCP_Cubic

Figure 10: TCP Cubic with a large receive buffer size(more than 3 MB) on the Verizon EVDO network. Itshows the severity of the bufferbloat problem.

0 20 40 60 80 100 1200

20

40

60

80

100

120

Time (Second)

Congestion W

indow

Siz

e (

KB

yte

s)

TCP_Cubic

Figure 11: TCP Cubic when the wireless link is de-graded. The congestion window shows the fast recoveryprocess.

as usual.Evaluating TCP Variants on cellular networks

Using a TCP bulk transmission test as an example, weuse MobiBed to setup a TCP sender on a Linux serverand a TCP receiver on a Samsung Galaxy Nexus phonelinked to the Verizon EVDO network. Once the phoneconnects to the server, the server starts sending data tothe phone. This type of test is typically used to measurethe download speed of a particular cellular networks. InMobiBed, we can easily change the TCP variant run-ning by changing a single line in the script, e.g., Line11 in Figure 6. Figure 9 shows the cwnd curves of dif-ferent TCP variants in one of the tests. The TCP Cubicand TCP NewReno show the same pattern, called “flatTCP”,4 previously observed by Jiang et al. [23, 22] un-der similar conditions. The reason is that the receive win-dow size (short for awnd) limits the increase of cwnd. Inthis test, we set tcp rmem max to 110208 bytes whichis the maximum of awnd. When cwnd is larger thanawnd, cwnd stay at a static value. Setting a small value oftcp rmem max is a typical but sub-optimal way used incurrent cellular networks to avoid the bufferbloat prob-

4Flat TCP is the phenomenon where the TCP congestion windowgrows to a static value and stays there until the session ends.

0 50 100 150 200 250 300 3500

10

20

30

40

50

60

70

Time (Second)

Co

ng

estio

n W

ind

ow

Siz

e (

KB

yte

s)

CWND

DRWA

Figure 12: DRWA Example 1. DRWA maintains a stablereceive window size when the link quality is stable

0 50 100 150 2000

20

40

60

80

Time (Second)

Co

ng

estio

n W

ind

ow

Siz

e (

KB

yte

s)

CWND

DRWA

Figure 13: DRWA Example 2. DRWA decreases the re-ceive window size when the link becomes degraded.

lem.5 If we do not limit the grow of cwnd in cellularnetworks, as shown in Figure 10, the cwnd increases tomore than 800 KB of packets in flight which is clearlybeyond the reasonable range of BDP (bandwidth delayproduct) in the EVDO network. This matches what Jianget al. [22] have observed as well.Monitoring TCP State Changes Figure 11 shows thecwnd’s reaction to TCP state changes during fast recov-ery mode. When the link quality turns bad, the TCP pro-tocol enters the fast recovery mode. The first two fastrecoveries successfully recover the lost packet (the cwndis not empty). We then see that the congestion windowinflates to enable fast recovery, and when it is completeit deflates to the half of the original congestion windowsize and then the cwnd returns control to the CUBIC al-gorithm. However, for the last recovery, it fails to recoverthe lost packets, probably due to multiple packet losses,before the timeout. When timeout, it drops to the bottomand increase the congestion window in slow start mode.Evaluating a New Protocol

Since all protocol modules are in the user space andwritten in Java, researchers can easily implement their

5Bufferbloat, termed by Gettys in 2010, is an unexpected phe-nomenon prevalent in the Internet where excessive buffering of packetswithin the network results in unnecessarily large end-to-end latency,jitter as well as throughput degradation in certain scenarios [22].

9

own protocols in MobiBed. Even more convenient, theycan simulate their protocols first in J-Sim and then testthe same implementations in MobiBed afterwards. Forexample, we implement the Dynamic Receive WindowAdjustment (DRWA) [23] algorithm to tackle bufferbloatin cellular networks. This corresponds to around 100lines of Java code in the current TCP implementation.To enable it during testing, one line in the script, e.g., !tcp setDRWA true is sufficient. In contrast, to test itdirectly on Android, one would have to modify the ker-nel code, compile the customized kernel, and then installit into a rooted phone.

Figure 12 shows that during a relative stable cellularlink, DRWA can maintain a stable state. Furthermore, asshown in Figure 13, when we artificially weaken the cel-lular network, e.g., use a metal box to gradually cover thephone, DRWA automatically decreases the advertised re-ceive window size to avoid bufferbloat when the windowsize is stationary.

In summary, we note that modifying protocols in Mo-biBed is much safer than in the Linux kernel. For exam-ple, in this test, the server we used is also running otherservices, e.g., group website, svn server, etc. If we didthe same test in the kernel, all the parameters changedabove in the server would apply to the other services aswell. The performance of these services would be re-duced when running a sub-optimal TCP variant. Evenworse, direct protocol modifications in the kernel onsmartphones could have a negative impact on the smart-phone’s performance which could result in draining thedata plan. Fortunately, MobiBed limits the consequencesof protocol modifications and have no direct impact onother applications running on the phones.

6 Conclusion

We proposed an open and scalable testbed, MobiBed, toenable network protocol testing and evaluation on mobilephones in the wild. By utilizing the UDP socket to em-ulate the raw network access socket we enable networkprotocols to run in user space instead of kernel space,thereby avoiding the need for superuser privileges or anymodifications to the kernel. We showed that for cellu-lar network bandwidths up to 4 Mbit/s, the throughputand latency overhead observed from a MobiBed TCPimplementation was negligible compared to the nativeTCP implementation. Also, through several exampleswe showed how MobiBed can facilitate network proto-col development and evaluation.

References

[1] JTcl Project. http://jtcl.kenai.com/.

[2] Mobile Marketing Statistics 2012.https://snaphop.com/2012-mobile-marketing-statistics/.

[3] Mobiperf 2.0. http://www.mobiperf.com/.

[4] Pluggable congestion avoidance modules.http://lwn.net/Articles/128681/.

[5] Tcpdump home page. http://www.tcpdump.org/.

[6] A. Botta, R. Canonico, G. Di Stasi, A. Pescape,G. Ventre, and S. Fdida. Integration of 3G con-nectivity in PlanetLab Europe: a step of an evolu-tionary path towards heterogeneous large scale net-work testbeds. Mob. Netw. Appl., 15(3):344–355,June 2010.

[7] M. Carbone and L. Rizzo. Dummynet revisited.ACM SIGCOMM Computer Communication Re-view, 40(2):12–20, 2010.

[8] G. Chatzimilioudis, A. Konstantinidis,C. Laoudias, and D. Zeinalipour-Yazti. Crowd-sourcing with Smartphones. IEEE InternetComputing, 16(5):36–44, Sept. 2012.

[9] B. Chun, D. Culler, T. Roscoe, A. Bavier, L. Peter-son, M. Wawrzoniak, and M. Bowman. PlanetLab:an overlay testbed for broad-coverage services.SIGCOMM Comput. Commun. Rev., 33(3):3–12,July 2003.

[10] C. Cornelius, A. Kapadia, D. Kotz, D. Peebles,M. Shin, and N. Triandopoulos. AnonySense:Privacy-Aware People-Centric Sensing. In Pro-ceedings of the 6th international conference on Mo-bile systems, applications, and services, MobiSys’08, pages 211–224, New York, NY, USA, 2008.ACM.

[11] E. Cuervo, P. Gilbert, B. Wu, and L. Cox. Crowd-Lab: An architecture for volunteer mobile testbeds.In Communication Systems and Networks (COM-SNETS), 2011 Third International Conference on,pages 1 –10, jan. 2011.

[12] T. Das, P. Mohan, V. N. Padmanabhan, R. Ramjee,and A. Sharma. PRISM: platform for remote sens-ing using smartphones. In Proceedings of the 8thinternational conference on Mobile systems, appli-cations, and services, MobiSys ’10, pages 63–76,New York, NY, USA, 2010. ACM.

[13] Y. Degani and S. Degani. iPerf for An-droid. http://www.cs.technion.ac.il/˜sakogan/-DSL/2011/projects/iperf.

10

[14] P. Dini, M. Portoles-Comeras, J. Mangues-Bafalluy, L. Dai, and S. Addepalli. A real-time cel-lular system architecture to experiment with UMT-S/HSDPA in a laboratory. In Testbeds and Re-search Infrastructures for the Development of Net-works Communities and Workshops, 2009. Trident-Com 2009. 5th International Conference on, pages1 –10, april 2009.

[15] Y. Dong, D. You, and J. Lui. Composition of Java-based router elements and its application to gener-alized video multicast. Network, IEEE, 18(6):27–33, 2004.

[16] D. Ely, S. Savage, and D. Wetherall. Alpine: Auser-level infrastructure for network protocol de-velopment. In Proceedings of the Third USENIXSymposium on Internet Technologies and Systems(USITS01), 2001.

[17] H. Falaki, D. Lymberopoulos, R. Mahajan, S. Kan-dula, and D. Estrin. A first look at traffic on smart-phones. In Proceedings of the 10th ACM SIG-COMM conference on Internet measurement, IMC’10, pages 281–287, New York, NY, USA, 2010.ACM.

[18] S. Ha, I. Rhee, and L. Xu. CUBIC: a new TCP-friendly high-speed TCP variant. ACM SIGOPSOperating Systems Review, 42(5):64–74, 2008.

[19] J. Huang, C. Chen, Y. Pei, Z. Wang, Z. Qian,F. Qian, B. Tiwana, Q. Xu, Z. Mao, M. Zhang,et al. Mobiperf: Mobile network measurement sys-tem. Technical report, University of Michigan andMicrosoft Research, 2011.

[20] J. Huang, F. Qian, A. Gerber, Z. M. Mao, S. Sen,and O. Spatscheck. A close examination of per-formance and power characteristics of 4G LTE net-works. In Proceedings of the 10th internationalconference on Mobile systems, applications, andservices, MobiSys ’12, pages 225–238, New York,NY, USA, 2012. ACM.

[21] J. Huang, Q. Xu, B. Tiwana, Z. M. Mao, M. Zhang,and P. Bahl. Anatomizing application performancedifferences on smartphones. In Proceedings of the8th international conference on Mobile systems,applications, and services, MobiSys ’10, pages165–178, New York, NY, USA, 2010. ACM.

[22] H. Jiang, Z. Liu, Y. Wang, K. Lee, and I. Rhee.Understanding bufferbloat in cellular networks. In

Proceedings of the 2012 ACM SIGCOMM work-shop on Cellular networks: operations, challenges,and future design, pages 1–6. ACM, 2012.

[23] H. Jiang, Y. Wang, K. Lee, and I. Rhee. Tacklingbufferbloat in 3G/4G networks. In Proceedings ofthe 2012 ACM conference on Internet measurementconference, pages 329–342. ACM, 2012.

[24] E. Kohler, R. Morris, B. Chen, J. Jannotti,and M. Kaashoek. The Click modular router.ACM Transactions on Computer Systems (TOCS),18(3):263–297, 2000.

[25] M.-R. Ra, B. Liu, T. F. La Porta, and R. Govindan.Medusa: a programming framework for crowd-sensing applications. In Proceedings of the 10th in-ternational conference on Mobile systems, applica-tions, and services, MobiSys ’12, pages 337–350,New York, NY, USA, 2012. ACM.

[26] L. Ravindranath, A. Thiagarajan, H. Balakrishnan,and S. Madden. Code in the air: simplifying sens-ing and coordination tasks on smartphones. In Pro-ceedings of the Twelfth Workshop on Mobile Com-puting Systems Applications, HotMobile ’12, pages4:1–4:6, New York, NY, USA, 2012. ACM.

[27] C. Shepard, A. Rahmati, C. Tossell, L. Zhong, andP. Kortum. Livelab: measuring wireless networksand smartphone users in the field. SIGMETRICSPerform. Eval. Rev., 38(3):15–20, Jan. 2011.

[28] A. Smith. Cell Internet Use 2012. Technical report,Pew Internet & American Life Project, April 2012.

[29] H. Soroush, N. Banerjee, A. Balasubramanian,M. Corner, B. Levine, and B. Lynn. Dome: A di-verse outdoor mobile testbed. In Proceedings ofthe 1st ACM International Workshop on Hot Top-ics of Planet-Scale Mobility Measurements, page 2.ACM, 2009.

[30] H. Tyan. Design, realization and evaluation of acomponent-based compositional software architec-ture for network simulation. PhD thesis, The OhioState UniverSity, 2002.

[31] H. Tyan, A. Sobeih, and J. Hou. Towards compos-able and extensible network simulation. In Paral-lel and Distributed Processing Symposium, 2005.Proceedings. 19th IEEE International, pages 8–pp.IEEE, 2005.

11