ADON: Application-Driven OverlayNetwork-as-a-Service for Data-Intensive Science

Sripriya Seetharam, Prasad Calyam, Tsegereda BeyeneUniversity of Missouri-Columbia, USA; Cisco Systems, USAEmail: {ssn68, calyamp}@missouri.edu, tbeyene@cisco.com

Abstract—Campuses are increasingly adopting hybrid cloudarchitectures for supporting data-intensive science applicationsthat require “on-demand” resources, which are not alwaysavailable locally on-site. Policies at the campus edge for handlingmultiple such applications competing for remote resources cancause bottlenecks across applications. These bottlenecks can beproactively avoided with pertinent profiling, monitoring andcontrol of application flows using software-defined networkingprinciples. In this paper, we present an “Application-drivenOverlay Network-as-a-Service” (ADON) that can manage thehybrid cloud requirements of multiple applications in a scalableand extensible manner using features such as: programmable“custom templates” and a “virtual tenant handler”. Our solutionapproach involves scheduling transit selection and traffic engi-neering at the campus-edge based on real-time policy controlthat ensures predictable application performance delivery formulti-tenant traffic profiles. We validate our ADON approachwith an implementation on a wide-area overlay network testbedacross two campuses, and present a workflow that eases theorchestration of network programmability for campus networkproviders and data-intensive application users. Lastly, we presentan emulation study of the ADON effectiveness in handlingtemporal behavior of multi-tenant traffic burst arrivals usingprofiles from a diverse set of actual data-intensive applications.

I. INTRODUCTION

Data-intensive applications in research fields such as bioin-formatics, climate modeling, particle physics and genomicsgenerate vast amounts of data that need to be processedwith real-time analysis. The general data processing facilitiesand specialized compute resources do not always reside atthe data generation sites on campus, and data is frequentlytransferred in real-time to geographically distributed sites (e.g.,remote instrumentation site, federated data repository, publiccloud) over wide-area networks. Moreover, researchers shareworkflows of their data-intensive applications with remotecollaborators for multi-disciplinary initiatives on multi-domainphysical networks [1].

Current campus network infrastructures place stringentsecurity policies at the edge router/switch and install firewallsto defend the campus local-area network (LAN) from potentialcyber attacks. Such defense mechanisms significantly impactresearch traffic especially in the case of data-intensive scienceapplications whose flows traverse wide-area network (WAN)paths. This has prompted campuses to build Science DMZs(de-militarized zones) [1] with high-speed (1 - 100 Gbps)programmable networks to provide dedicated network infras-tructures for research traffic flows that need to be handled inparallel to the regular enterprise traffic.

This work was supported by the National Science Foundation under awards:ACI-1246001 and ACI-1245795, and Cisco Systems. Any opinions, findings,and conclusions or recommendations expressed in this publication are thoseof the author(s) and do not necessarily reflect the views of the NationalScience Foundation or Cisco Systems.

Fig. 1: Illustration to show the need for application perfor-mance visibility and control over a wide-area network path

The advanced network infrastructure components in Sci-ence DMZs that help with high-performance networkingto remote sites and public clouds include: (i) software-defined networking (SDN) based on programmable OpenFlowswitches [2], (ii) RDMA over Converged Ethernet (RoCE)implemented between zero-copy data transfer nodes [3] fordata transport acceleration, (iii) multi-domain network per-formance monitoring using perfSONAR active measurementpoints [4], and (iv) federated identity and access managementusing Shibboleth-based entitlements [5].

However, if multiple applications accessing hybrid cloudresources compete for the exclusive and limited Science DMZresources, the policy handling of research traffic can causea major bottleneck at the campus edge router and impactthe performance across applications. Figure 1 illustrates anactual problem scenario we faced when we initiated a filetransfer as part of a research application (Advanced DataTransfer Service) using RoCE protocol on the high-speedoverlay between the University of Missouri (MU) and TheOhio State University (OSU) [6]. As shown, the achievedtransfer time was substantially low compared to the expectedtheoretical transfer time. Upon investigation, we discoveredthat our application’s traffic was being rate limited to 2 Gbpsat OSU edge router even though the link’s capacity was capableof 10 Gbps speeds. Since RoCE protocol assumed a 10 Gbpslink availability and is highly sensitive to packet loss, ourapplication performance suffered severely.

As evident from the above scenario, there is a need toprovide dynamic Quality of Service (QoS) control of Sci-ence DMZ network resources versus setting a static ratelimit affecting all applications. The dynamic control shouldhave awareness of research application flows with urgent orother high-priority computing needs, while also efficientlyvirtualizing the infrastructure for handling multiple diverseapplication traffic flows. The virtualization obviously shouldnot affect the QoS of any of the provisioned applications,and also advanced services should be easy-to-use for data-intensive application users, who should not be worrying aboutconfiguring underlying infrastructure resources.

Fig. 2: Multi-tenant application workflows with Layer-2 over-lays and Layer-3 routing on a shared wide-area infrastructure

Our work in this paper aims to solve the network vir-tualization problem at campus-edge networks using dynamicqueue policy management for individual flows, while makingnetwork programmability related issues a non-factor for data-intensive application users. More specifically, we present a new“Application-Driven Overlay Network-as-a-Service” (ADON)architecture to intelligently provision on-demand network re-sources by performing a direct binding of applications toinfrastructure and provide fine-grained automated QoS control.

The novelty and contributions of our work are as follows:we detail how ‘network personalization’ can be performedusing a concept of “custom templates” to catalog and handleunique profiles of application workflows. We also detail amulti-tenant architecture for real-time policy control of an‘Overlay Network-as-a-Service’ through a “Virtual TenantHandler” (VTH). The VTH leverages awareness of the overall‘load state’ at the campus edge, and the individual application‘flow state’ using software-defined performance monitoringintegration within the overlay network paths. Using the customtemplates and VTH concepts, ADON can manage the hybridcloud requirements of multiple applications in a scalableand extensible manner. It ensures predictable application per-formance delivery by scheduling transit selection (choosingbetween regular Internet or high-performance Science DMZpaths) and traffic engineering (e.g., rate limit queue mappingbased on application-driven requirements) at the campus-edge.

We validate our ADON implementation on a wide-areaoverlay network testbed across OSU and MU campuses con-nected with Internet2 Advanced Layer2 Service (AL2S), andpresent a workflow that we adopted to ease the orchestration ofnetwork programmability for both campus network providersand data-intensive application users. Lastly, we present a de-tailed emulation study of the ADON effectiveness in handlingtemporal behavior of multi-tenant traffic burst arrivals usingprofiles from a diverse set of actual data-intensive applicationsused in research and education use cases.

The remainder paper organization is as follows: Section IIpresents related work. Section III details custom templates forexemplar application workflows. Section IV describes ADON’smodular architecture with VTH components. Section V de-scribes ADON implementation on an actual testbed, and in anemulation study. Section VI concludes the paper.

Fig. 3: Sequential steps of ADON during on-demand resourceprovisioning for a data-intensive application flow

II. RELATED WORK

Existing works such as [7], [8] and [9] recognize similarapplication-driven network virtualization issues at network-edges, and have proposed new architectures based on SDNprinciples. In [7], application level requirements are pro-grammed using an inter-domain controller implemented usingOpenFlow, and a custom-built extensible session protocol isused to provide end-to-end virtual circuits across campusDMZs. In [8] that is closely related to our work, QoS pa-rameters are programmed dynamically based on high-levelrequirements of different kinds of application traffic. Theauthors argue (similar to our argument in context of Figure 1)that there is a need for dynamic QoS configuration based onapplication needs, and current practice of manual configurationby network administrators hinders application performance.In [9], the authors propose a new controller design for QoSbased routing of multimedia traffic flows.

In contrast, our work in this paper is focused on the require-ments and challenges at the campus-edge. We handle ScienceDMZ resources and personalize network configurations (e.g.,rate limit queue mappings) that need to be controlled inreal-time for meeting hybrid cloud computing needs of data-intensive application flows.

III. NETWORK PERSONALIZATION FOR APPLICATIONSWITHIN ADON

Figure 2 shows how data-intensive applications can co-existon top of a shared wide-area physical infrastructure topology,with each application demanding local/remote network orcompute resources with unique end-to-end QoS requirements.In order to effectively handle the diverse QoS requirementsat the campus edge router, several challenges need to beaddressed. One of the important challenges is the need tohave ‘application performance visibility and control’ within theprovisioned resources for individual applications. This requiresmaintaining a catalog of application profiles in terms ofResource Specifications (RSpecs) and Quality Specifications(QSpecs). In addition, policies should be determined for theextent to which programmable capabilities at the campus edgecan be used to ‘personalize’ the network overlay setup basedon: (a) the individual application RSpecs and QSpecs, and(b) the temporal behavior of multi-tenant traffic burst arrivals.

In the following subsections, we first describe the conceptof custom templates that can be used within ADON to developa catalog of application profiles. Following this, we apply thecustom template concept for exemplar data-intensive applica-tion workflows with diverse QoS requirements.

(a) Neuroblastoma application (b) RIVVIR application

(c) GENI classroom experiments application (d) ElderCare-as-a-Service application

Fig. 4: Topologies of various data-intensive applications that need network personalization

A. Custom Templates

Our concept of custom templates within ADON is similarto the best practices such as Amazon Web Services (AWS)Machine Image (AMI) [10] and RSpecs in the NSF-supportedGlobal Environment for Network Innovations (GENI) [11].Works such as [12] also suggest the value of using templatesthat can allow for composing and executing workflow pipelinesfor data-intensive applications in a reusable manner.

Figure 3 shows how custom templates can be used as partof the sequential steps of ADON auto-orchestration during on-demand resource provisioning for a data-intensive applicationflow that needs an overlay network path. The details of thesteps in ADON orchestration are as follows: First, a researcherof a data-intensive application can securely request the ADONby authenticating with a Federated Identity and Access Man-agement (Federated IAM) system that uses Shibboleth-basedentitlements [5]. Such Federated IAM systems are necessaryto handle multi-institutional policy specifications pertaining tocases such as: (i) How can a data-intensive application userat Campus A be authenticated and authorized to reserve HPCresources or other scientific instruments at a remote CampusB? or (ii) How can a OpenFlow controller at one campusbe authorized to provision flow spaces within a backbonenetwork in an on-demand manner? (iii) Who can subscribeto the performance measurements related to a data-intensiveapplication to monitor workflow status and track/troubleshootany bottleneck anomaly events?

Subsequently, the researcher provides his/her data inten-sive application handling specifications through a simple andintuitive application dashboard mashup. The specifications caninclude details such as destination host (i.e., remote collab-orator or remote instrument site) and application type (e.g.,remote interactive volume visualization, video streaming, filetransfer or compute resource reservation). Next, the applicationspecifications are subsequently matched to a custom templatewith RSpecs and QSpecs that closely match the applicationtype for discovery/reservation of the necessary compute andnetwork resources.

The custom template can be pre-configured by a Perfor-mance Engineer to apply specific resource descriptions andassociated campus policies that can be interpreted by e.g., anetwork flowvisor (i.e., proxy for OpenDaylight or POX [14])to instantiate flows on intermediate OpenFlow switches, and bya compute hypervisor to instantiate virtual machines within a

data center. We refer to Performance Engineer as the one whoserves as the primary ‘keeper’ and ‘helpdesk’ of the ScienceDMZ equipment, and the success of this role is in the techni-cian’s ability to augment traditional System/Network Engineerroles on campuses and serve high-throughput computing needsof researchers.

In addition to helping the Performance Engineer with theresource configuration, custom templates also help in config-uring real-time network performance monitoring within theoverlay network path to provide the application performancevisibility that can be used to define triggers for dynamicresource adaptation. Moreover, performance bottlenecks suchas those observed in Figure 1 can be avoided through useof custom templates, and in exception cases where end-to-endQoS configuration is not possible, bottlenecks can be relativelymore easily discovered and overcome.

B. Applications Workflows

1) Neuroblastoma Data Cutter Application: As shown inFigure 4(a), the workflow of the Neuroblastoma application [6]consists of a high-resolution microscopic instrument on a localcampus site (at MU) generating data-intensive images thatneeds to be processed in real-time to identify and diagnoseNeuroblastoma (a type of cancer) infected cells. The process-ing software and HPC resources required for processing theseimages are available remotely at OSU, and hence images fromMU need to be transferred in real-time to the remote OSUcampus. To handle the highly large scale data transfers, theapplication relies on advanced file transfer protocols such asRoCE and GridFTP technologies that support parallel TCPflows between the two campuses. A corresponding Neurob-lastoma application template can be given as: (i) RSpec -parallel TCP flows with high bandwidth bursts, and (ii) QSpec- high flow throughput with no packet loss and high flowpriority to provide fast-enough application response time for ahistopathological evaluator.

2) Remote Interactive Volume Visualization Application(RIVVIR): As shown in Figure 4(b), the RIVVIR applica-tion [13] at OSU deals with real-time remote volume visu-alization of 3D models of small animal imaging generatedby MRI scanners. When such an application needs to beaccessed for remote steering and visualization by thin-clients,the network path between the two sites should have as muchavailable bandwidth as possible. A corresponding RIVVIR

Fig. 5: ADON reference architecture

application template can be: (i) RSpec - Layer 2 networksteering (over Internet2) of application traffic, and (ii) QSpec -Low latency/jitter flow with high bandwidth and medium flowpriority to help with interactive analysis with a thin-client.

3) GENI Classroom Lab Experiments: As shown in Fig-ure 4(c), a class of 30 students conducting lab experiments atMU in a Cloud Computing course [14] require resources acrossmultiple GENI racks. As part of the lab exericses, multipleVMs need to be reserved and instantiated on remotely locatedGENI racks. There can be a sudden burst of application trafficflows at campus edge router, especially the evening before thelab assignment submission deadline. A corresponding GENIClassroom application template can be: (i) RSpec - Multiplelayer 2 flows to remote sites with low bandwidth requirements,and (ii) QSpec - Low packet loss and medium flow priorityto allow students to finish the lab exercises.

4) ElderCare-as-a-Service Application: As shown in Fig-ure 4(d), the ElderCare-as-a-Service application [14] consistsof an interactive video streaming session between a therapiston MU campus and a remotely residing elderly patient at aKansas City residence for performing physiotherapy exercisesas part of telehealth interventions. During a session, the qualityof application experience for both users is a critical factor(especially in skeletal images from Kinect sensors), and theapplication demands strict end-to-end QoS requirements tobe usable. A corresponding ElderCare-as-a-Service applicationtemplate can be: (i) RSpec - Deterministic flow path with high-resilience for network link failures on Layer 3 (regular Internetwith Google Fiber last-mile), and (ii) QSpec - Consistenthigh available bandwidth with very low or no jitter and highflow priority to allow an elderly patient to closely follow thepostures being exercised in the session.

IV. ADON ARCHITECTURE

In this section, we describe the policy implementationarchitecture of ADON that leverages the custom templates forfine-grained QoS control and customization of programmablecompute and network resources. Figure 5 shows the ADONarchitecture, which consists of the application, middleware andthe infrastructure layers within which individual componentsinteract with each other to provision resources for incomingapplication requests.

Fig. 6: Individual components of the Virtual Tenant Handler

The high-level application requirements along withRSpecs, QSpecs and application priority are captured inthe application layer. Depending upon the resources beingrequested in the infrastructure layer, and the campus policyrules (maintained by the Performance Engineer), the routingand queue policy assignments are applied in the middle-ware layer for each application being provisioned. Real-timeperformance monitoring of individual flows can be used toconfigure adaptation triggers within already provisioned flows,or even to reject a new application flow if the required QoSlevels cannot be met given the load of already provisionedflows. Such middleware layer functions can be implementedwith: (i) Control Module, (ii) Network Flowvisor, and (iii)Compute Hypervisor. In this paper, we mainly focus on theControl Module’s ‘Custom Template Catalog’ and NetworkFlowvisor module’s ‘Virtual Tenant Handler’ (highlighted inred in Figure 5) that are necessary to implement ADON, andthe Compute Hypervisor issues are beyond this paper scope.

A. Custom Template Catalog

The Control Module consists of the Template Generatorcomponent which exposes RESTful APIs for configuring ap-plication type, application priority and routing specificationsthat can be programmed within the application layer. TheTemplate Generator module also allows the Performance En-gineer to save a successfully configured template in a customtemplate catalog database, which allows re-use for futureflow provisioning instances. The QoS and application priorityparameters are then fed into the Network Flowvisor moduleby programming the required REST APIs such as: queuepolicies and bandwidth requirements. The Federated IAMcomponent within the Control Module features an ‘entitlementservice’ module for all campuses that federate their ScienceDMZ infrastructures using third-party frameworks such as theInternet2 Incommon federation [15]. It also allows for centrallymanaging entitlements based on mutual protection of privacypolicies between institutions to authorize access to differentmulti-domain infrastructure components.

B. Virtual Tenant Handler

Virtual Tenant Handler (VTH) is responsible for dynami-cally handling incoming application flows and providing the in-telligence for adaptive network resource allocation. As shownin Figure 6, the Policy Engine interacts with the TemplateGenerator component on the top layer for accepting the RSpecand QSpec parameters along with application priority withinthe custom templates. The Policy Engine then interacts with theFlow Scheduler to check the priority of the new application

with the existing applications. The Routing Controller is re-sponsible for deciding on which port number of the OpenFlowswitch should the application flow be provisioned for trafficsteering such as Layer 2 (Science DMZ path over Internet2AL2S) or Layer 3 (regular Internet).

If Science DMZ path is selected for the application flow,the Dynamic Queue Manager is responsible for provisioningthe right QoS policies. To accomplish such right provisioning,we utilize the minimum rate and maximum rate properties ofqueue configuration on OpenFlow switches as provided by theOpenFlow 1.3 specification [16]. The configured queues onthe OpenFlow switch ports are then mapped to incoming ap-plication flows using the set-queue action set of the OpenFlowspecification. As shown in Figure 7, the queue slots are mappedbased on the application priority as specified in the high-level application requirements. A higher priority applicationis mapped to a queue with the maximum available bandwidth.

In case the desired queue is not available, it is mappeddown the queue priority level to be provisioned in the next bestavailable priority queue. The mapping is done until a queueslot that can provide an acceptable QoS is selected. If found,the flow is provisioned on the selected queue, or else the flowis pushed to the Flow Scheduler component. However, for amedium priority application, if Layer 2 slot is not available andthe QoS parameters are not constrained, the flow is directed toa Layer 3 default queue. If none of the slots are available, theflow can be pushed again to the Flow Scheduler to be retrievedlater once the appropriate queue is available. The DynamicQueue Manager then interacts with the Resource Aggregator toupdate the available resources once a given flow is provisionedon its overlay network.

The VTH also monitors the load states at each queue andflow states of each application using ‘Flow State Monitor’ and‘Load State Monitor’ components. These components receiveinputs from the ‘Network Performance Measurement’ modulewhich is a real-time SDN monitoring system that providesnetwork health status notifications such as flow statisticsand other such QoS impacting factors. The module providesoptions to reconfigure existing paths based on the customtemplate directed resource reservations. In case the load statein terms of number of applications contending for Layer 2service creates a link saturation, VTH can steer the traffic onLayer 3 if acceptable QoS parameters can allow a push of thenew flows into the Flow Scheduler.

The VTH further interacts with the underlying OpenFlowcontroller for installing the required flows on the OpenFlowswitches. The ‘Network Topology Manager’ (part of the Open-Flow controller) implements shortest available path algorithmswith weighted and non-weighted deterministic routes to remoteWAN-accessible sites and provides graphical abstractions tothe VTH module with topology link states. The algorithm usedin the VTH is formally discussed in the following section.

C. ADON Resource Provisioning Algorithm

As shown in Algorithm 1, a new application anew entersthe VTH module. If the net resource available RN where Nis the total number of applications within VTH, is greater thanthe threshold resource value Tr of the system, the applicationcan be pushed in Step-1 to a weighted fair-queue FIFOFlow scheduler. The threshold value Tr can be a parameterprogrammed by the Performance Engineer in the ApplicationLayer. In case there are available resources, the priority of thenew application Panew

is compared with the already scheduledapplication as priority Pas

. Whose ever priority is higher, the

Fig. 7: Illustration of dynamic queue management

slice sa is instantiated using the createSlice function withinthe Policy Engine.

Once the slice is created in Step-2, the optimal resource rafor the application is computed in Step-3 using the functioncomputeResource based on application template parameterssuch as destination point, RSpec (ar) and QSpec (aq). Notethat ra is a vector containing the information: (i) Which queuehas to be allocated for the application? (ii) Which port suchas Layer 2 or Layer 3 service has to be assigned for theapplication? (iii) What is the required bandwidth for the appli-cation? In Step-4, the computed resources are then subtractedfrom the net resource RN to indicate the new resource status.Suppose, after a certain timeout, the flow statistics providedby a certain flow implies under or no utilization of resourcesby the application; then in Step-5, the slice is deleted andresources are released into the available resource pool for otherscheduled application flows in the Flow Scheduler component.

Algorithm 1 ADON Resource Provisioning Algorithm

1: Input: Application flow anew2: Output: Resource allocation ra for application flow an3: begin procedure4: /*Step-1: Resource threshold check */5: if net resource RN < threshold value Tr then6: if scheduler not null && Pas > Panew then7: an = as8: else9: an = anew

10: end if11: /*Step-2: Application slice creation*/12: sa = createSlice(an)13: /*Step-3: Resource usage calculation*/14: ra = computeResource(sa, ar, aq)15: /*Step-4: Update available resources*/16: RN = R - ra17: else18: Push application flow to resource scheduler19: end if20: /*Step-5: Application termination or timeout*/21: RN = R + ra22: Pop existing application flow from scheduler queue and

repeat the above steps

The above logic ensures high priority flows receive band-width provisioning that is adequate to satisfy performancerequirements, and allows per-flow isolation when there aremultiple lower priority application flows competing for net-work resources. In addition, it ensures low priority flowsdo not experience high rejection rates, while simultaneouslymaximizing the network resource utilization.

Fig. 8: Collaborative Science DMZ testbed between two cam-puses for handling multi-tenancy of data-intensive applications

V. EXPERIMENTAL EVALUATION

In this section, we first describe a case study featuring val-idation experiments of our ADON implementation on a wide-area overlay network testbed across OSU and MU campusesconnected with Internet2 AL2S. Next, we present a detailedemulation study we conducted with application workflows toanalyze the VTH algorithm results while handling temporalbehavior of multi-tenant traffic burst arrivals.

A. Implementation Case Study

The testbed setup as shown in Figure 8 consists of OSU andMU campuses connected through an extended VLAN overlaythat involves an Internet2 AL2S connection by way of localregional networks of OARnet in Ohio, and GPN/MoreNet inMissouri, respectively. Each Science DMZ has a matchingDTN equipped with dual Intel E5-2660, 128GB of memory,300GB PCI-Express solid state drive, and dual Mellanox 10Gbps network cards with RoCE support. Each Science DMZhas perfSONAR measurement points for continuous monitor-ing at 1 - 10 Gbps network speeds. A common Dell R610 nodein the OSU Science DMZ is used to run the VTH module alongwith OpenDaylight OpenFlow controller that controls both theOSU and MU Science DMZ OpenFlow switches.

We ran an experiment by providing the application require-ments such as the source site, and destination site - along withapplication type, which is a real-time image transfer for pro-cessing of medical images within Neuroblastoma application- by following the steps illustrated previously in Figure 3.The application requirements and policy specifications wereconfigured as high bandwidth, Layer 2 routing with priorityqueue assignment in the corresponding custom template. Thecompute instances on Owens cluster at OSU were reserved,including 6 running GPU nodes and memory space of 3000MB at both ends. We also introduced tcp cross traffic usingIperf tool to simulate a parallel flow along with researchapplication flow.

These specifications were then mapped by the VTH moduleto lower-layer control policies in order to instantiate the flowrules on the OpenFlow switches at both campus edges. Oncethe flow rules were in place with our ADON implementation,the data flow was automatically instantiated on the infrastruc-ture layer with desired QoS and no manual intervention. Theresearch application flow was instantiated on Layer 2 (Internet2domain) while the cross traffic was steered on Layer 3 (regularInternet) by the OpenFlow switch from MU end. The RoCEtraffic was detected based on the ether type 0x8915 and Iperf as0x800. A 500 MB image file was transferred from MU to OSU

Fig. 9: Timeline of a campus edge router handling multipledata-intensive application workflows

Owens cluster in 5.13 seconds. The time taken to process theimage was about 4 seconds (i.e., compute time) on the clusterand the transfer of processed image back to MU took around5.2 seconds (i.e., communication time) aggregating to a totalof 14.4 seconds of “compute plus communication” time withinthe ADON provisioned path.

B. Emulation Study

We used Mininet network emulator for VTH experimentsthat involved synthetic traffic flows using the “tc” and “iperf”network utility tools. Figure 9 shows the timeline, from timet1 to t7 as seen by an edge OpenFlow router/switch handlingapplication workflows, as they enter and exit the VTH module.RIVVIR application workflow (see Figure 4(b)) was initiatedas a UDP flow at time t1 with a guaranteed bandwidth of 10Mbps (typical requirements of remote desktop access with rawencoding) and latency of 50 ms (RTT between OSU and MU).At time t2, ElderCare-as-a-Service application workflow (seeFigure 4(d)) was started as a new UDP flow with a guaranteedbandwidth of 100 Mbps (typical requirement of a Kinect videostream) and latency of 30 ms (RTT between MU and KansasCity).

At this moment, the Dynamic Queue Manager within theVTH instantiated the queues on the OpenFlow switch toprovide the required QoS guarantees for each of the concurrentflows. The total jitter observed when both flows coexistedon the link are captured with and without VTH dynamicqueue management in Figure 10(a). We can see that the jitteris significantly reduced (improved performance) for both theapplications, especially for the video flow using the VTHmodule with application-specific queue policies. This is dueto the fact that the VTH reduced the external fragmentation ofavailable bandwidth caused by static policy management on theswitch ports. The dynamic queue mapping of the UDP flowsto the requested bandwidth provided performance isolation toeach of the flows, and hence reduced their jitter QoS metricvalues.

At time t3, GENI Classroom workflow (see Figure 4(c))was started as a TCP flow with burst traffic pattern (burst rate -1 Mbps and buffer size- 10 KB similar to web traffic) parallelto the RIVVIR workflow. Both flows co-existed without affect-ing each other’s QoS policies as both flows were assigned theirindividual priorities on the queues. At time t4, Neuroblastomaworkflow (see Figure 4(a)) was started as a parallel TCP flow

(a) Jitter comparison

(b) Throughput comparison

Fig. 10: Results of dynamic queue assignment by VTH

with 5 parallel streams to simulate a GridFTP application forimage file transfer. VTH then triggered the dynamic queueconfiguration for assigning a prioritized bandwidth of 600Mbps for Neuroblastoma and 360 Mbps bandwidth for GENIClassroom experiments (on a 1 Gbps link).

Figure 10(b) shows throughput of the two workflowsachieved with and without VTH. Bandwidth is equally splitwhen there is no dynamic queue management for both flows.However with VTH, internal fragmentation of bandwidth isreduced. The total bandwidth is sliced between the two flows asper their individual priorities ensuring each flow is only utiliz-ing the requested bandwidth as provided in the application QoStemplates. This slicing happens until time t5 when the GENIClassroom flow exits and resources are released. However,when a new RIVVIR workflow starts again at time t6 while theNeuroblastoma application is currently provisioned, the newflow is rejected and pushed to the Flow Scheduler. This isbecause the new flow’s QoS requirements cannot be guaranteedand mapped in the Dynamic Queue Manager. This scenariooccurs due to the resource unavailability of the link which isfully utilized by the prioritized data-intensive Neuroblastomaapplication flow.

Thus, we can conclude from the above experiments thatthe VTH is effective in scheduling transit selection and trafficengineering at the campus-edge based on real-time policycontrol that ensures predictable application performance de-livery, when handling temporal behavior of multi-tenant trafficburst arrivals corresponding to a diverse set of data-intensiveapplications.

VI. CONCLUSION

In this paper, we presented a novel ADON architecture withan application-driven overlay network-as-a-service approach tosupport multi-tenant data-intensive application flows with hy-brid cloud resource needs. With the pent-up resource require-ments of data-intensive application flows, traditional network

infrastructures are not scalable or flexible for effectively han-dling such flows, especially in cases with urgent or real-timecomputing requirements. Using our ADON architecture, weshowed that the application-specific policies can be effectivelycontrolled at the campus edge based on individual applicationflow requirements, and the ‘friction’ imposed due to firewallsfor enterprise traffic flows can be overridden for data-intensivescience applications.

The novelty of our work is in our approach for ‘networkpersonalization’ that can be performed using a concept of“custom templates” that helps a Performance Engineer tocatalog and handle unique profiles of application workflows inan automated and repeatable manner. We also presented designdetails and validation experiments of a multi-tenant architec-ture featuring a “Virtual Tenant Handler” (VTH) for real-timepolicy control of an ‘Overlay Network-as-a-Service’ withina campus Science DMZ environment with high-performancenetworking capabilities such as OpenFlow switches and RoCE-based data transfer nodes. Further, we demonstrated how ourADON architecture and implementation were capable of pro-viding predictable performance to data-intensive applications,without any changes to existing campus network infrastructuredesigned for regular enterprise traffic.

Our future work includes integrating multiplegeographically-distributed campuses to the ADON architectureapproach as a community model, and conducting additionalwide-area overlay network and emulation experiments.

REFERENCES

[1] E. Dart, L. Rotman, B. Tierney, M. Hester, J. Zurawski, “The ScienceDMZ: A Network Design Pattern for Data-Intensive Science”, Proc. ofIEEE/ACM Supercomputing, 2013.

[2] N. McKeown, T. Anderson, H. Balakrishnan, et. al., “OpenFlow: En-abling Innovation in Campus Networks”, ACM SIGCOMM ComputerCommunication Review, Vol. 38, No. 2, 2008.

[3] P. Lai, H. Subramoni, S. Narravula, A. Mamidala, D. K. Panda, “De-signing Efficient FTP Mechanisms for High Performance Data-Transferover InfiniBand”, Proc. of ICPP, 2009.

[4] A. Hanemann, J. Boote, E. Boyd, et. al., “perfSONAR: A ServiceOriented Architecture for Multi-Domain Network Monitoring”, Proc. ofService Oriented Computing, 2005.

[5] R. Morgan, S. Cantor, S. Carmody, W. Hoehn, K. Klingenstein, “Feder-ated Security: The Shibboleth Approach”, EDUCAUSE Quarterly, 2004.

[6] P. Calyam, A. Berryman, E. Saule, H. Subramoni, P. Schopis, G.Springer, U. Catalyurek, D. K. Panda, “Wide-area Overlay Networking toManage Accelerated Science DMZ Flows”, Proc. of IEEE ICNC, 2014.

[7] I. Monga, E. Pouyoul, C. Guok, “Software-Defined Networking for BigData Science”, Proc. of IEEE/ACM Supercomputing, 2012.

[8] W. Kim, P. Sharma, J. Lee, S. Banerjee, J. Tourrilhes, S. Lee, P. Yalagan-dula, “Automated and Scalable QoS Control for Network Convergence”,Proc. of INM/WREN, 2010.

[9] H. Egilmez, S. Dane, K. Bagci, A. Tekalp, “OpenQoS: An OpenFlowcontroller design for multimedia delivery with end-to-end Quality ofService over Software-Defined Networks”, Proc. of APSIPA ASC, 2012.

[10] Amazon Web Services - http://aws.amazon.com[11] M. Berman, J. Chase, L. Landweber, A. Nakao, M. Ott, D. Raychaud-

huri, R. Ricci, I. Seskar, “GENI: A federated testbed for innovativenetwork experiments”, Elsevier Computer Networks Journal, 2014.

[12] D. Gunter, L. Ramakrishnan, S. Poon, G. Pastorello, V. Hendrix, D.Agarwal, “Designing APIs for Data-Intensive Workows: Methodologyand Experiences from Tigres”, IEEE e-Science, 2013.

[13] P. Calyam, A. Berryman, A Lai, M. Honigford, “VMLab: Infrastruc-ture to Support Desktop Virtualization Experiments for Research andEducation”, VMware Technical Journal, 2012.

[14] P. Calyam, S. Seetharam, R. Antequera, “GENI Laboratory ExercisesDevelopment for a Cloud Computing Course”, Proc. of GENI Researchand Educational Experiment Workshop, 2014.

[15] Internet2 InCommon - https://incommon.org[16] OpenFlow Switch Specification - https://www.opennetworking.org