Jupiter: A Networked Computing ArchitecturePradipta Ghosh, Quynh Nguyen, Pranav K Sakulkar, Aleksandra Knezevic, Jason A. Tran, Jiatong Wang

Zhifeng Lin, Bhaskar Krishnamachari, Murali Annavaram, and Salman Avestimehr

Abstract—In the era of Internet of Things, there is an increas-ing demand for networked computing to support the require-ments of the time-constrained, compute-intensive distributedapplications such as multi-camera video processing and datafusion for security. We present Jupiter, an open source networkedcomputing system that inputs a Directed Acyclic Graph (DAG)-based computational task graph to efficiently distribute the tasksamong a set of networked compute nodes regardless of theirgeographical separations and orchestrates the execution of theDAG thereafter. This Kubernetes container-orchestration-basedsystem supports both centralized and decentralized schedulingalgorithms for optimally mapping the tasks based on informationfrom a range of profilers: network profilers, resource profilers,and execution time profilers. While centralized scheduling al-gorithms with global knowledge have been popular among thegrid/cloud computing community, we argue that a distributedscheduling approach is better suited for networked computingdue to lower communication and computation overhead in theface of network dynamics. To this end, we propose and implementa new class of distributed scheduling algorithms called WAVE onthe Jupiter system. We present a set of real world experimentson two separate testbeds - one a world-wide network of 90 cloudcomputers across 8 cities and the other a cluster of 30 Raspberrypi nodes, over a simple networked computing application calledDistributed Network Anomaly Detector (DNAD). We show thatdespite using more localized knowledge, a distributed WAVEgreedy algorithm can achieve similar performance as a classicalcentralized scheduling algorithm called Heterogeneous EarliestFinish Time (HEFT), suitably enhanced for the Jupiter system.

I. INTRODUCTION

With the miniaturization of hardware in the era of Internetof Things (IoT), the presence of economical low-compute-power edge devices such as cell phones, car dashboard, anddrones have become ubiquitous near end users. This hasopened up the domain of edge or fog computing [1] thatfocuses on exploiting all the devices near end users to complywith the skyrocketing demand for computationally intensiveapplications such as image processing and voice recognitiontowards autonomy and personalized assistance. Interestingly,a significant subset of these cutting-edge time-constrained,compute-intensive distributed applications rely on an orderlyprocessing of the streaming data generated from a set ofdevices that maybe geographically dispersed. This brings us

All authors are with the Ming Hsieh Department of Electrical Engineering,University of Southern California, Los Angeles, CA, USA.

Corresponding Email: pradiptg@usc.eduThis material is based upon work supported by Defense Advanced Research

Projects Agency (DARPA) under Contract No. HR001117C0053. Any views,opinions, and/or findings expressed are those of the author(s) and should notbe interpreted as representing the official views or policies of the Departmentof Defense or the U.S. Government.

to the newly emerging field of Networked Computing orDispersed Computing that focuses on a joint optimizationof computation and communication costs to distribute theexecution of a Directed Acyclic Graph (DAG) based task graphamong a network of compute nodes that may be geograph-ically distributed. Networked Computing can be thought ofas a mixed architecture between Edge Computing and CloudComputing where the network of compute nodes might containeither or both edge processors and cloud-based processors.This new field of Networked computing calls for a distributedsystem that can optimally leverage the available computeresources in a network of compute nodes while accountingfor any network delays that may impact the timely processingof data.

In this paper, we present Jupiter, an open-source systemfor networked computing that contains the necessary toolsto efficiently map the tasks from a task-DAG into a setof geographically distributed networked compute processors(NCPs) with the main focus on ‘Makespan’ minimization. Wedefine the ‘Makespan’ of the DAG as the time required togenerate an output via executing the entire task DAG on oneset of input files or input chuck of data. Jupiter also administersthe actual processing of the tasks along with efficient datatransfer between them. Jupiter relies on the reputed open-source container-orchestrator tool from Google called Kuber-netes [2] for implementation of the three main components:(1) Profilers that gather statistics about the network condition,resource availability in the NCPs, and execution time ofthe tasks on the NCPs, (2) Task Mapper that leverages theinformation available from three different types of profilermodules to optimally schedule or map the tasks into theNCP nodes to minimize the Makespan of the DAG, and (3)CIRCE that boots-up the tasks according to the task mappingand administers the task executions and data transfers. Fortask to NCP mapping, Jupiter has plug-n-play type provisionsfor both centralized and decentralized mapping or schedulingalgorithms. While centralized mappers such as the Hetero-geneous Earliest Finish Time (HEFT) [3] are proved to bepromising and efficient for cloud/grid computing, we arguethat a distributed scheduler with comparable performance ismore appropriate for networked computing systems. To thisend, we propose the WAVE framework that is a novel class ofdecentralized task mapping algorithms which we demonstrateto have similar empirical statistics as the HEFT algorithm.

To test the performance of the Jupiter system for varyingnetwork and resource conditions, we perform a wide range of

arX

iv:1

912.

1064

3v1

[cs

.DC

] 2

3 D

ec 2

019

experiments on two fairly large testbeds: (1) a 90 node testbedbased on the DigitalOcean cloud platform where we hand-picked the servers from a set of 8 geographically distributedcluster locations, and (2) a 30 node in-house Raspberry Pi3(RPI3) cluster connected via a Cisco switch to control thenetwork characteristics. For the experiments, we have alsoimplemented a sample networked computing application calledthe Distributed Network Anomaly Detection (DNAD) whichis focused on real-time defence against Distributed Denialof Service (DDoS) attacks in a network. Our experimentsshow that for highly resource-constrained devices, such as theRaspberry Pi3, the original HEFT performs quite poorly. Thisled us to design a modified version of HEFT that has betterperformance in such resource-constrained devices.

In summary, our contributions in this paper can be listed asfollows:• We propose a novel open-source networked computing

system called the Jupiter that supports proper profilingof the resources, efficient centralized/decentralized task-to-compute node mapping for an application-DAG, andadministered execution of the application-DAG.

• We propose a new class of distributed local-information-based scheduling/mapping algorithms called WAVE thathas similar performance to a well-known centralized,globally informed heuristic called the HEFT.

• We formulate a new application for networked comput-ing called the Distributed Network Anomaly Detection(DNAD) which is focused on real-time defence againstDistributed Denial of Service (DDoS) attacks in a net-work.

• Based on a range of experiments, we discover someshortcomings of the HEFT like algorithms on the Jupitersystem and, thereafter, propose some constructive modi-fications.

Fig. 1: Illustration of the DAG based Networked Computing problem.The black lines denote communication links, the red lines denote themapping, and the blue lines denote data flows in the DAG.

II. PROBLEM DESCRIPTION

First of all, let us assume that we have a network ofN heterogeneous networked compute processors (NCPs) thatare geographically distributed across the world. Therefore,the end-to-end latency between these NCPs are statistically

different as well as dynamic. Let’s say, we are interested indeploying an application-DAG that consists of T tasks wherethe input sources are distributed across the world. Now, thegoal is to properly map the tasks from the application DAGto the NCPs such that the Makespan of the application-DAGis minimized. The Makespan in this context would dependon both the compute powers of the NCPs chosen as wellas the delays on the network paths between the NCPs. Foran illustration, refer to Fig. 1 where the application DAGconsists of 6 tasks with two geographically separated inputsources. Now, the goal is to optimally map the tasks on thegeographically distributed NCPs such that the output can bemade available the fastest.

A. The DNAD Application

We present a sample application for Networked Computingcalled the Distributed Networked Anomaly Detector (DNAD).The main goal of this application is to use a network ofcomputation capable routers to detect Distributed Denial ofService (DDoS) attacks. We assume that a set of border routersof a distributed administrative network monitors the incomingtraffic to periodically generate traffic statistics which can beemployed to detect the potentially anomalous IP addresses andto take affirmative actions against such IP addresses. One caneasily have a powerful central server to collect the data andprocess it. However, for a geographically dispersed network,the communication delay between different border routers andthe cloud is significant which makes the process non-realtimeas well as costly due to the usage of cloud resources. Thenetworked computing based solution can come to the rescue byusing all the compute resources available on the network, suchas the processors on the routers, towards realtime processingof the traffic statistics and realtime protection of the networkagainst such DDoS attacks. The task DAG for the DNAD ispresented in Fig. 2. Next, we briefly explain each of the tasks.

Fig. 2: The DAG for the DNAD Application

Local Processing: The local processing task node is mainlythe observation point of the network which monitors theincoming traffic and collects the statistics by hashing the trafficstatistics using the IP addresses. Next, it splits the input data

stream into three independent streams based on the evaluatedhash values. Each stream is further sent to one of the threechild tasks i.e., aggregation points (illustrated in Fig. 2).

Aggregation Point: In our DNAD implementation, we havethree aggregate task nodes that collect statistics with the simi-lar hash value from the observation points i.e, local processingunits. In this example, we have only one observation point forsimplicity. Ideally, the number of aggregation nodes can beany positive number. This number is mainly used for loadbalancing the detection process. The output of each aggregatenode is sent to a range of detector node.

Anomaly Detectors: Any network anomaly detector isbound to have false alarms and missed detections. Usingseveral detectors in parallel and combining their outputs to-gether is warranted to provide more robustness to the detectionperformance. We use two different detectors on each data-stream: (1) Simple detector, which simply applies a thresholdon the traffic generated from each IP in order to filter out theanomaly and (2) Astute detector, which is an implementationof the Astute anomaly detector [4]. One can easily extend thisframework to use more than 2 types of detectors.

Fusion Center: Each fusion center aggregates and com-bines the output of the respective anomaly detectors to outputa unified list of anomalous IP addresses for that particular datastream.

Global Fusion: The global fusion unit combines the outputsof the fusion centers to output a file with the final inferenceabout the observed traffic.

III. THE JUPITER ARCHITECTURE

Jupiter is a networked computing system to automate themapping of application DAG to an arbitrary network underboth centralized and decentralized settings. The Jupiter systemconsists of three main modules: Profiler, Task Mapper, andthe CIRCE dispatcher. The inputs to the Jupiter consist ofthe Directed Acyclic Task Graph (DAG) information, the taskfiles, and the information (such as IP or node name) aboutavailable compute nodes. The profiler module of the Jupiterconsists of three different types of profilers: (1) NetworkProfiler that maintains statistics about the bandwidth and end-to-end delay between the available NCPs, (2) Resource Profilerthat profiles the resource availability of each NCP in terms ofCPU and Memory availability, and (3) Execution profiler thatprofiles the execution time of each task of the DAG in each ofthe available NCPs. The information from the profilers and theinput files are fed to the task mapper module which outputs amapping of the DAG tasks into the available NCPs based onthe mapping algorithm used. Next, the generated task-to-NCP-mapping is used by the CIRCE dispatcher to dispatch the taskson respective NCPs, monitor the input-output of each taskto administer the respective task execution, and transfer thedata/files between consecutive tasks of the DAG. In the currentJupiter system, we have provision for two different classes oftask mappers: (1) Centralized HEFT and (2) DecentralizedWAVE. A Jupiter configuration file is used for choosingbetween these different options of task mappers as well as

setting a range of parameters to customize for application-specific requirements. In the Jupiter architecture, we assumethat there exists at least one NCP in the network that can actas an administrative node to the network which we refer toas the “Home NCP”. The Home NCP can be any randomlyselected NCP of the network if no such distinct administrativeNCP exists. In Fig. 3, we illustrate the architecture of theproposed Jupiter system along with the data flow betweendifferent modules.

Fig. 3: The Jupiter Architecture

A. Profilers

To provide support for a broad range of mapping algorithms,Jupiter’s profiler module consists of three different types ofprofilers: Network, Resource, and Execution Profiler. A keyarchitectural component of all three types of profilers is thatthere exists one Home profiler which runs on the Home NCP,while rest of the NCPs run a copy of the Worker Profiler. Thisis illustrated in Fig. 4. The Home profiler acts as a masterto initiate and orchestrate the profiling process while keepingtrack of available NCP information. The Worker profilersperform the actual profiling job on each NCP which we discussin the following.

1) Network Profiler: A major component in the Makespanof a DAG-based application is the file transfer latency betweenconsecutive tasks in the task graph. Thus, the end-to-end delaybetween two NCP nodes is an important parameter for taskmapping. The network profiler in the Jupiter system providesthat information by having a network profiling job run oneach node, which we also refer to as the Worker NetworkProfiler, and periodically probing the network. To this end,each Worker profiler periodically sends a randomly generatedfile with known file size to each of the other compute nodesvia the well-known file transfer protocol called Secure Copy(SCP). The file transfer times are recorded and curve-fit usinga quadratic regression with respect to the file-size (f ) as:l = p+ q · f + r · f2 where p, q, r are empirically determinedconstants. We opted for a quadratic fit as it is the empiricalbest fit towards approximate file transfer time for varying fil-sizes.

Fig. 4: Illustration of the Jupiter deployed system. There exists aHome NCP that runs all the Home Profilers, the Home Task Mapper,and Home CIRCE. The Home CIRCE is used purely for experimentalpurpose. Similarly, all the other NCPs in the networks run the Workerparts of the Profilers, WAVE (if used), and CIRCE.

2) Resource Profiler: The resource profiler provides twotypes of information to the task mappers: CPU availabilityand storage availability. Each of the Worker Resource Profilerruns a Flask server to listen for resource profiling requests andreplies with the current CPU and storage usage upon receivinga request. Moreover, each worker profiler periodically sendsresource profiling request to every other NCP in the networkand stores the collected statistics in a local database. There isalso a provision for immediate polling of resource informationwhere the Worker Profiler returns the current informationinstead of last known information. This is done by instantpolling of other NCPs upon receiving such requests from thetask mapper or the application.

3) Execution Profiler: For optimal allocation of tasks, thetask mappers such as HEFT might need information aboutthe execution times of the individual tasks for each of theNCPs. HEFT like task mappers do not use the raw CPU usagestatistics available from the Resource Profilers. To supportsuch requirements, we have the third and final type of profiler:the Execution Profiler. However, the complete execution timeinformation is available only after the tasks are executed oneach of the available NCPs. To make this information avail-able even before the tasks are actually mapped, the WorkerExecution Profiler on each NCP runs the entire DAG withsome sample input files and sends the statistics to the HomeExecution Profiler. Moreover, the Home Execution Profileralso collects information about runtime statistics once the tasksare mapped and executed via CIRCE.

B. Task Mapper

The task mapper module of the Jupiter is the most importantmodule of the Jupiter system. As the name suggests, themain function of this module is to optimally map individualtasks of a task-DAG into a set of available compute nodes(NCPs) such that the Makespan of the task DAG is minimized.

To this end, there are two classes of approach that can beopted for: centralized and decentralized. In the centralizedapproach, a central node gathers the global information of thenetwork of compute nodes from the profilers and leveragesthis information for optimal placement of tasks. On the otherhand, a distributed approach leverages the local profilinginformation in each of the compute nodes for task placements.In the current version of the Jupiter, we have made availabletwo different classes of task mappers: centralized HEFT anddecentralized WAVE.

1) HEFT: Heterogeneous Earliest Finish Time(HEFT) ([5], [3]) is a well-known heuristic in grid/cloudcomputing for mapping a directed acyclic task graphinto a network of heterogeneous compute nodes that alsoaccounts for the communication times between the nodes.HEFT operates in a sequence of two phases: ranking andprioritization, and processor selection. In the first phase, i.e.,ranking or prioritization phase, HEFT defines a priority ofeach task ti as follows:

ranku(ti) = ωi + maxtj∈succ(ti)

(ci.j + ranku(tj)) (1)

where the subscript “u” refers to “upwards rank” which isdefined as the expected distance of the task from the end of thecomputation, ti refers to task i, ωi is the average computationcost of the task i among all the compute nodes, ci,j refersto the average communication cost of the data communicatedbetween task ti and tj for all pairs of compute nodes, andsucc(ti) refers to the set of dependent tasks in the DAG. Forexample, the set of dependent tasks for task C in Fig. 5a,succ(C), is {D, E}.

In the second phase i.e., the processor selection phase,HEFT assigns the tasks to the NCPs based on the rankscalculated in the ranking or prioritization phase. In eachiteration of the task assignment, HEFT picks the task whichhas the highest priority and has all the dependent tasks alreadymapped. Next, HEFT schedules the task on an NCP that willminimize the earliest finish time of that task. This processcontinues until all the tasks are mapped. Finally, HEFT outputsthe overall task to NCP mapping along with a timeline tofollow for the executions.

2) WAVE: While centralized task mappers are appropriatefor cloud computing like scenarios with a network of ge-ographically neighboring compute nodes, a distributed taskmapper is more appropriate for Networked Computing dueto lower communication and computation overhead as well asfast reaction time. To this end, we propose a new class ofdecentralized task mapper algorithm called the WAVE. Beforedetailing WAVE, let us define the notion of “task controller”which is an NCP that is in charge of mapping a particular setof tasks from the DAG. In the WAVE architecture, there existsa coordinator or home WAVE node (which runs on the HomeNCP as illustrated in Fig 4) that initiates the whole process,while rest of the nodes, which we refer to as the worker WAVEnodes, perform the actual mapping in a distributed fashion.The WAVE algorithm works in two phases as follows.

Task Controller Selection: In this phase, the WAVE homenode chooses a unique “task controller” for each task of theDAG. For the first level of tasks (e.g., task A and B for thetask DAG presented in Fig. 5a), the home node itself acts asthe task controller. For the rest of the tasks, the home nodechooses the task controller as follows.• Iterate over the tasks from the DAG in their topological

orders. For the sample DAG presented in Fig. 5a, onetopological order would be {A, B, C, D, E, F}.

• For each non-input task, check if any of its parent tasks(Tasks A and B are the parent tasks to task C in Fig. 5a)are already controllers.

• If one of the parents is already a task controller, thenappoint that parent as the controller for this task.

• If no parent is already a task controller or multiple parentsare task controllers, then choose the parent task with thesmaller topological index as the parent.

Note that, so far we refer to tasks as task controllers insteadof NCPs because at this stage of WAVE the tasks are notmapped/bound to any particular NCP. In the next step ofWAVE, we explain how we map the tasks to the NCPs andconsequently map the task controllers to the NCPs. For illus-tration purpose, let us assume that the task controller selectionoutput for Fig. 5a isMT = {Home→ {A,B}, A→ C,C →{D,E}, D → F} (as shown in Fig. 5b). Here A→ C impliesthat task A is the task controller for task C.

(a) (b)

Fig. 5: (a) A Sample Task DAG (b) WAVE Illustration for Task DAGpresented in Fig. 5a. The square boxes represent the task controllers.

Task Mapping: In this phase, the WAVE algorithm mapsthe tasks into appropriate NCPs. The Home node kick-startsthis process by determining the NCPs for each of the inputtasks in the DAG, according to the geographical location of thedata source. E.g., for the DAG presented in Fig. 5a, the WAVEhome will place task A and task B on two NCPs that are neardata source 1 and 2, respectively, as presented in Fig. 5b.Note that we assume the data source locations to be known.Once the process completes, the WAVE home broadcasts thismapping to the respective NCPs. Next, the NCPs of thealready-mapped tasks will perform similar mapping for thetasks they are in charge of. For example, the NCP of task Awill decide where to run task C which thereafter decides where

to run tasks D and E. This process continues until all the tasksare mapped. Every time a new task mapping is complete, thehome node is informed by the respective task controller. Oncethe whole process is complete, the WAVE Home returns themapping information to the next component of Jupiter: theCIRCE dispatcher. This process is illustrated in Figure 5b.

Now, a task controller chooses the optimal NCPs for thetasks by following two different logic as follows.

Random WAVE:This is the simplest version of WAVE where the task con-

trollers randomly select a NCP from the list of available NCPs.The task controller does not incorporate the communicationand computation costs in the mapping logic which makes themapping completely non-optimized. This is used as a baselinealgorithm and proof of the concept for WAVE.

Greedy WAVE:The greedy WAVE is a complex version of WAVE that

incorporates the profiling information for mapping tasks tothe NCPs. In the greedy WAVE, each of the task controllersconnects to the local profilers to get network and resourcestatistics (it does not use execution profilers). Next, each taskcontroller NCP, say NCP i, follows a sequence of operationto map the tasks controlled by it.• Based on the end-to-end latency statistics from the

network profiler, find the minimum delay dimin =minj di,j∀j 6= i and i, j ∈ 1, 2, · · · , N where N is thenumber of NCPs.

• Use the calculated dimin to filter a feasible neighboringcompute node set, Si

d = {j : di,j < dth} where dth =k · dimin is the threshold latency. We have empiricallychosen an value of k = 15 for our experiments due to avery wide distribution of network delay.

• Use the resource information i.e., the CPU usage, pj ,and the memory usage mj of each neighbor j to rankthe neighbors in Si

d as follows:

rank(i, j) = ωd · di,j + ωp · pj + ωm ·mj ∀j ∈ Sid (2)

where ωd, ωp, ωm are three weighing constants todetermine the rank. We empirically choose a value ofωd = ωp = ωm = 1/3 for the experiments presented inthis paper.

• Use the rank information to map the tasks. If the taskcontroller is responsible for n tasks and n ≤ |Si

d|, itmaps the tasks to n top ranked neighbors based on theordering of the tasks on the DAG. Otherwise, first mapthe top |Si

d| tasks on the |Sid| neighboring NCPs (one-

to-one) according to the rank and task ordering. This isfollowed by repeating the same process for rest of thetasks.

C. The CIRCE Dispatcher

The CIRCE dispatcher is the third and the final moduleof the Jupiter system. The CIRCE dispatcher is the part ofJupiter that inputs the task-to-NCP mapping and dispatchesthe tasks on the respective nodes. CIRCE wraps the task codesto support an input-output queuing system. CIRCE creates an

input folder/queue and an output folder/queue for each taskand takes care of transferring the output of a task to the inputthe next task in the DAG using the well-known SCP tool.Every time a new data file arrives at the input folder, CIRCEstarts the execution of the respective task. Sometimes, a taskmight require the output of more than one parent task as itsinput. In such cases, CIRCE also takes care of waiting forall the inputs to arrive before starting the execution. At thecompletion of execution, CIRCE starts the file transfer processto the next task. If there are multiple child tasks, CIRCEtransfers a copy of the output to the input of each of the childtasks. CIRCE uses a sequence number for the ordering of theinput data.

IV. IMPLEMENTATION DETAILS

Jupiter is implemented on top of a well-known open-sourcecloud-based-container-orchestrator from Google called the Ku-bernetes ([6], [2]). Before explaining the implementation, letus briefly introduce some key concepts such as Container,Docker, and the Kubernetes architecture.

A. Containers, Docker, and Kubernetes

The most common norm in cloud computing today is touse Virtual Machines (VMs) to support the demand of userswhile keeping necessary isolation between the user processesrunning on the same physical machine. VMs run inside aguest OS and accesses the guest Hardware via the concept ofvirtual Hardware. To support this sort of isolation from the realhardware as well as among different VMs, the computationaloverhead of VM is substantial. This limits the number ofconcurrent VMs on a Physical machine to a very small numbersuch as 3-5 for a standard Quadcore desktop with 12GB RAM.Moreover, due to the lack of direct access to the Hardware,the functionality of VMs are restricted. Containers, on theother hand, are the most cutting-edge convention for processorvirtualization that provides isolations similar to traditionalVMs but with much less computing power requirements.Unlike VMs, a container image is a lightweight standaloneexecutable that includes all the requires modules, libraries,codes, and tools to run it. A container directly runs on theguest OS and is considered as a single process by the guestOS. All the processes inside a container are viewed as a sub-process of the main process. Because of this low computationrequirement, one can run hundreds of containers on a physicalmachine. The concept of a container has been around fora while but was not popular until the advent of a specifictype of containers called the Dockers [7] in mid-2014. Amongthe handful of available container orchestration tools, GoogleKubernetes, Apache Mesos, and Docker Swarm are the mostpromising ones. Out of these options, we opted for Kubernetesdue to its popularity as well as its unique features suchas a support of Raspberry Pi3 devices. Briefly speaking, inKuberenetes, there exists one central high power master node(which we refer to as the K8 Master) that maintains a networkof compute nodes, keeps tracks of the deployed containers, andrestarts the deployed containers in case of failures. Note that

the K8 Master is different from the Home NCP required forJupiter system which can be run on the same NCP or differentNCPs. A detailed overview of Kuberenetes can be found inthe official website https://kubernetes.io/https://kubernetes.io/.

B. Jupiter on Kubernetes

We implemented each component of the Jupiter in Dockersto support parallelism, have isolation between different Jupitermodules, support scalable and easy administrations, and sup-port multiple simultaneous DAGs. Moreover, by using con-tainers, every module of Jupiter is uniquely addressable (viaunique IP and port numbers) which helps in a uniform systemimplementation. Another reason behind the Dockerization ofJupiter is to make it compatible with the Kubernetes system.Next, we briefly detail different types of Dockers used in theJupiter system.

Network and Resource Profiler Dockers: For compact-ness, we put the network and resource profiler inside oneDocker instead of separate ones. We combine network homeprofiler and resource home profiler into a combined Dockerwhich runs on the Home NCP. Similarly, we combine theworker network profiler and worker resource profiler into asingle Docker which runs on each NCP of the network exceptthe Home NCP.

Execution Profiler Dockers: Due to different functionalitythan the network and resource profilers, we have kept theexecution profilers in a separate Docker. Again, we create twodifferent types of execution profiler Docker to correspond tothe home execution profiler (runs on the Home NCP) and theworker execution profiler (runs on all NCPs expect the HomeNCP), respectively.

HEFT Profiler Docker: Because HEFT is a centralizedprofiler, there is only one Docker needed for HEFT which canrun on any NCP of the network. However, for consistency, wechoose to run it on the Home NCP.

WAVE Dockers: For both WAVE Greedy and WAVE Ran-dom, we have the notion of home and worker. Therefore, weneed two separate Dockers for “WAVE-home” and “WAVE-worker” that runs on the Home NCP and the rest of the NCPs,respectively.

CIRCE Dockers: The experimental implementation ofCIRCE has the notion of home and worker as well. Here,CIRCE home is used mainly to emulate a data source aswell as to collect different statistics whereas a CIRCE workerDocker follows the description presented in Section III-C.Therefore, we have two Dockers for CIRCE as well: “CIRCE-home” and “CIRCE-worker”. The CIRCE worker Dockercontains all the task files but can run only one task of theDAG at a time. Thus, the number of CIRCE worker Dockerson the network equals the number of tasks in the task-DAG. Ifan NCP has multiple tasks allocated to it, it will run multipleCIRCE Dockers.

V. EXPERIMENTAL RESULTS AND ANALYSIS

We analyze the performance of the Jupiter system via arange of experiments with the DNAD application. For these

experiments, we use two different clusters with Kubernetes.The first cluster consists of 90 Virtual machines, also referredto as Droplets, from a cloud provider called the Digital Ocean.Out of the 90 VMs, 13 VMs have 2GB RAM while the resthave 3GB RAM. Each of these VMs has 20 GB of disk spaceavailable. We handpicked the set of VMs from 8 availablegeographic locations across the world. The geographic distri-bution of nodes is presented in Fig. 6a. We use an 8GB VM asthe Kubernetes master node for this cluster and for the HomeNCP, we randomly pick one from the 90 VMs.

The second testbed consists of 30 in-house Raspberry Pi3nodes with 1GB RAM and 64GB SD cards, as illustrated inFigure 6b. They are connected via a Cisco switch to control thenetwork conditions and topology. For the Kubernetes clusterimplementation, we have opted for a mixed architecture wherean AMD64 Virtual Machine with 8GB RAM and 10GB diskspace acts as the Kubernetes master while all the actualcompute nodes are ARM32 real processors on the RaspberryPis. Again, the Home node is chosen at random from the 30RPIs.

(a) (b)

Fig. 6: (a) Geographical Distribution of the Digital Ocean VMs and(b) the RPI Cluster

In both testbeds, we ran the Jupiter system with all threetypes of task mappers i.e., HEFT, WAVE Random, and WAVEGreedy. For each configuration of Jupiter, we ran the wholeJupiter system 25 times. In each run, once the CIRCE deploy-ment is complete, we feed a sequence of 10 pre-loaded fileswith file sizes in the range of 10KB to 300KB and recorddifferent statistics such as the Makespan of the task DAG andthe execution times of individual tasks. Note that, we refer toHEFT as original HEFT in this section as we propose somemodification to the HEFT based on the experiment results toimprove its performance.

A. DAG Makespan Analysis

In the first set of experiments, we compare the Makespanstatistics of the DNAD application for different task mappingalgorithms. In Fig. 7a, we present the Makespan statistics forthe RPI cluster for 10 sequential inputs files which shows thatamong the original HEFT, WAVE Greedy, and WAVE Randomalgorithm, the performance of the WAVE Greedy is the best,followed by the performance of WAVE random and HEFT.While the bad performance of the WAVE random is prece-dented due to the random nature of the mapping algorithm,

the bad performance of the original HEFT was not warrantedsince HEFT is a well-respected heuristic for DAG based taskgraph in cloud/grid computing. Upon further investigations,we find that this bad performance of HEFT is rooted at theprocessing power and stability limitations of the RPIs. We findthat resource-constrained devices like RPI3 are very unstableand failure prone if we run too many CIRCE containers(more than 3-4 CIRCE containers) alongside with the otherrequired components (i.e, the profilers and task mappers) ofthe Jupiter. The original HEFT works under the assumptionthat the task executions will follow the timeline suggested bythe algorithm. However, to support the continuous executionof incoming files and parallel processing in the NCPs, Jupiterkeeps a separate CIRCE docker running for each of the tasks.When HEFT tries to optimize the Makespan by reducingcommunication overhead and putting many tasks on the sameNCP, it ends up overloading the RPIs. While the Jupiter systemcan recover from failures, multiple failures of the overloadedRPIs actually ends up adding more delay in the executionof the tasks as well as the communication between tasksdue to temporary disruptions of the data flow. To circumventthis issue, we propose a minor modification to the originalHEFT where HEFT is restricted to allocate no more than cmcontainers per NCP where the number cm is dependent uponthe processing power of the node. We empirically choose avalue of cm = 2 for the RPI3 cluster. We will refer to thisversion of HEFT as the modified HEFT. The performanceanalysis of the modified HEFT (presented in Fig. 7a) showsthat, by this slight modification, the performance of HEFT isimproved and becomes comparable to the WAVE Greedy.

We perform a similar set of experiments on the DigitalOcean cluster and present the results in Fig. 7b. Figure 7bshows that the performance of HEFT Original and the WAVEGreedy are comparable while the performance of the WAVERandom is the worst followed by the modified HEFT. Againthe bad performance of WAVE Random is due to randomselection of the NCPs without really accounting for anycommunication and processing overheads. The reason behindmodified HEFT not performing well is that a stable cloudsystem cluster like the Digital Ocean with higher computingpower can accommodate more than 2 CIRCE dockers per NCP.By putting the restriction on HEFT, we are forcing HEFTto choose a different NCP which thereby adds networkingdelay in the Makespan calculation. To verify this, we performanother set of experiments on the Digital Ocean cluster withmodified HEFT and cm = 4. The results presented in Fig. 7bshows that with cm = 4 the performance of modified HEFTis similar to original HEFT. This suggests that the modifiedHEFT have similar or better performance than the originalHEFT provided that cm properly account for the processorlimitations.

In summary, modified HEFT with a properly selected valueof cm and WAVE Greedy have the best performance amongall four choices of mappers (Original HEFT, Modified HEFT,WAVE Greedy, and WAVE Random) in Jupiter. The resultsalso substantiate that a distributed algorithm which relies

only on local information can have similar performance asa centralized globally-informed algorithm.

1 2 3 4 5 6 7 8 9 10File Number

0

50

100

150

200

Average

Makespanin

Seconds

Makespan statistics for 10 files (RPI Cluster)

HEFT Original

HEFT Modified (cm = 2)

WAVE Random

WAVE Greedy

(a)

1 2 3 4 5 6 7 8 9 10File Number

0

50

100

150

Average

Makespanin

Seconds

Makespan statistics for 10 files (Digital Ocean)

HEFT Original

HEFT Modified (cm = 2)

HEFT Modified (cm = 4)

WAVE Random

WAVE Greedy

(b)

Fig. 7: Makespan Statistics of the DNAD Task-DAG for 10 DifferentFiles in (a) the RPI Cluster and (b) the Digital Ocean Cluster.

B. Runtime Analysis of the Task Mapping Algorithms

We perform another set of experiments to analyze the run-time of the modified HEFT, Greedy WAVE, and the RandomWAVE algorithms. For these experiments, we do not considerthe original HEFT as the runtime of original HEFT is similarto the runtime of the modified HEFT. The runtime statistics arepresented on Table I. Table I shows that for the 30 node RPIcluster, the runtime of HEFT is the lowest and the runtimeof WAVE Random and WAVE Greedy are similar. Ideally,the HEFT algorithm runtime should be larger than WAVE asHEFT requires all the profiler data to be gathered at a centralpoint which takes a considerable amount of time whereasWAVE relies only on local information. Moreover, the amountof data to be processed in HEFT is larger than the amountof data to processed by each of the NCPs in WAVE. As areason behind this counter-intuitive result, we hypothesize thatfor a cluster of that size (30 nodes) the communication timeoverhead for sequential task assignments via task controllersover the network (as in WAVE) is much larger than the timerequired to gather all the statistics in a central location (asin HEFT). To verify this hypothesis, we run a similar setof experiments on the Digital Ocean Cluster with 30 nodes,60 nodes, and 90 nodes. The results presented in Table Ishows that as the number of nodes increases, the runtimeof HEFT gradually becomes comparable (for 60 nodes) and

even worse than Greedy WAVE (for 90 nodes). This suggeststhat with increasing network size, the time required to gatherall the statistics at a central location increases and eventuallysurpasses the communication time overhead for sequential taskassignments in WAVE.

Another interesting observation is that WAVE Randomruntime is actually slightly larger than the WAVE Greedywhereas intuitively the runtime of WAVE random should besmaller. Upon further investigation, we discovered that thisinconsistency between the expected result and the observedresult is due to the communication delay difference betweenthe task controllers. By randomly assigning the NCPs, the end-to-end delay in communication between the task controllers,which is required for the mapping purpose, becomes largercompared to the same for WAVE Greedy. We also discover thatthis delay is the largest component of the runtime of WAVE.For illustration, in the 90 node cluster, the time requiredto retrieve the network data is ≈ 0.05s and the resourcedata is ≈ 10s whereas the runtime of the WAVE Greedy isaround 100s. This suggests that there is still a lot of room forimprovement in the WAVE algorithm.

Lastly, it is evident from Table I that the runtime of theWAVE algorithm remains almost same regardless of the clustersize as it mainly depends on the number of tasks in the DAGrather than the actual number of NCPs. On the other hand,the performance of the HEFT algorithm varies proportionalto the number of NCPs in the network. This makes WAVEmuch more scalable than HEFT for a distributed networkedcomputing system.

TABLE I: Runtime Statistics of the Mapping Algorithms in Seconds

ClusterDetails

Modified HEFT Random WAVE Greedy WAVEMean STD Mean STD Mean STD

RPI Cluster(30 NCPs)

102.48 69.27 156.37 78.74 154.94 73.80

Digital Ocean(30 NCPs)

55.73 25.73 90.48 1.48 89.94 0.92

Digital Ocean(60 NCPs)

89.39 32.32 111.06 64.62 92.40 8.12

Digital Ocean(90 NCPs)

136.36 45.52 101.35 11.45 97.44 2.52

VI. RELATED WORKS

The wide range of related works mainly lies within therich literature of cloud computing, edge computing, and gridcomputing.

The well-known field of cloud computing deals with vir-tualization technology that enables data centers to providea range of services such as computing service and storageservice to the users on pay-per-use basis [8]. The typicalcloud computing architecture with hundreds of powerful co-located servers requires all the data to be first collected atthe cloud before the actual computation can be performed [9].In contrast, the Jupiter system enables geographical proximitybased mapping of the tasks without requiring the data to becollected at a central point.

There also exists some challenges of cloud computing suchas the data pre-processing from heterogeneous unstructured

data sources, intermittent connectivity of the data sources,and low latency requirement [10] that can be solved bothby edge computing tools [11] and networked computing tools(e.g., Jupiter). However, edge computing architectures has itsown set of limitations such as limited geographical span andlimited amount of resources [12]. In contrast, our proposedJupiter system considers a geographically dispersed set ofheterogeneous compute nodes (they can be both cloud nodesor edge nodes) and distributes the tasks among them whileminimizing the Makespan, if the application is represented asa DAG.

In the grid computing domain [13], there also exists someframeworks for mapping tasks on geographically distributedclusters such as Pegasus [14] and Falcon [15]. However,they assume a static and relatively well characterized networkwith simple applications for centralized task mapping. Incomparison, the Jupiter framework along with the WAVEscheduling algorithm is designed to support complete geo-graphical separation along with decentralized task mapping forany complex DAG based applications. Moreover, Jupiter canbe easily modified into a cloud computing, edge computing,or even a grid computing system, if required.

In the context of cloud computing, there exists a rangeof scheduling algorithms that schedules tasks from a DAGinto multiple cores of a single or multiple co-located proces-sors ([16], [17]) without accounting any communication costs.On the other hand, there exist some centralized schedulersfor DAG-based applications that do account for the commu-nication overhead between multiple processors such as theHeterogeneous Earliest Finish Time (HEFT) [3], the LongestDynamic Critical Path (LDCP) algorithm [18], the DynamicLevel Scheduling (DLS) [19], and the Critical Path On aCluster (CPOC) algorithm [3]. Nonetheless, these algorithmsare mainly aimed at geographically co-located processors withconsiderably low and static delay statistics. Our proposedWAVE algorithm, on the other hand, is designed mainly for asparse and dynamic network of compute nodes.

While, there also exists some state-of-the-art algorithms fordistributed scheduling ([20], [21], [22]), to our knowledge,only a few of them are directly applicable to DAG based taskscheduling on a networked computing cluster. Nonetheless, theJupiter system can support a suitably enhanced version of anyof these scheduling algorithms, if needed.

VII. CONCLUSION

In this paper, we proposed a new distributed system forNetworked Computing called the Jupiter that can efficientlydistribute and deploy tasks from a DAG based task graphwith the goal of Makespan minimization. We also proposed anew class of distributed task-mapping or scheduling algorithmscalled the WAVE that leverages local information of the NCPsfor distributed mapping of the tasks. Through a wide rangeof real-world experiments on a 90-node (distributed across 8cities in 7 countries) Cloud-based cluster and a 30-node edgecomputing cluster, we show that WAVE can perform similar

or even better than a globally informed centralized task-mapping algorithm called the HEFT. However, while WAVEshows promising attributes in terms of Makespan performanceand scalability, it lacks any optimality guarantee and thereis still a lot of room for improvements. Therefore, in ourfuture works, we would like to explore for an optimal settingsof WAVE. We would also like to explore more centralizedand decentralized mapping algorithms to find a better andmore generic trade-off between centralized and decentralizedalgorithms for Networked Computing. Lastly, we would like tointroduce the concept of mapping-algorithm-independent loadbalancing in Jupiter to support dynamics in incoming datatraffic without having to re-run the task Mapping algorithm.

REFERENCES

[1] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A surveyon mobile edge computing: The communication perspective,” IEEECommunications Surveys & Tutorials, vol. 19, no. 4, pp. 2322–2358,2017.

[2] K. Hightower, B. Burns, and J. Beda, “Kubernetes: Up and running diveinto the future of infrastructure,” 2017.

[3] H. Topcuoglu, S. Hariri, and M.-y. Wu, “Performance-effective andlow-complexity task scheduling for heterogeneous computing,” IEEEtransactions on parallel and distributed systems, vol. 13, no. 3, pp. 260–274, 2002.

[4] F. Silveira, C. Diot, N. Taft, and R. Govindan, “Astute: Detectinga different class of traffic anomalies,” ACM SIGCOMM ComputerCommunication Review, vol. 41, no. 4, pp. 267–278, 2011.

[5] H. Topcuoglu, S. Hariri, and M.-Y. Wu, “Task scheduling algorithms forheterogeneous processors,” in IEEE Heterogeneous Computing Work-shop (HCW), 1999.

[6] D. Bernstein, “Containers and cloud: From lxc to docker to kubernetes,”IEEE Cloud Computing, vol. 1, no. 3, pp. 81–84, 2014.

[7] D. Merkel, “Docker: lightweight linux containers for consistent devel-opment and deployment,” Linux Journal, vol. 2014, no. 239, p. 2, 2014.

[8] S. Singh and I. Chana, “A survey on resource scheduling in cloudcomputing: Issues and challenges,” Journal of grid computing, vol. 14,no. 2, pp. 217–264, 2016.

[9] H. T. Dinh, C. Lee, D. Niyato, and P. Wang, “A survey of mobilecloud computing: architecture, applications, and approaches,” Wirelesscommunications and mobile computing, vol. 13, no. 18, pp. 1587–1611,2013.

[10] I. A. T. Hashem, I. Yaqoob, N. B. Anuar, S. Mokhtar, A. Gani, andS. U. Khan, “The rise of big data on cloud computing: Review andopen research issues,” Information Systems, vol. 47, pp. 98–115, 2015.

[11] P. Mach and Z. Becvar, “Mobile edge computing: A survey on archi-tecture and computation offloading,” arXiv preprint arXiv:1702.05309,2017.

[12] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “Mobile edgecomputing: Survey and research outlook,” CoRR, vol. abs/1701.01090,2017.

[13] F. Berman, G. Fox, T. Hey, and A. J. Hey, Grid computing: making theglobal infrastructure a reality. John Wiley and sons, 2003, vol. 2.

[14] E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman,G. Mehta, K. Vahi, G. B. Berriman, J. Good et al., “Pegasus: Aframework for mapping complex scientific workflows onto distributedsystems,” Scientific Programming, vol. 13, no. 3, pp. 219–237, 2005.

[15] I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde, “Falkon: afast and light-weight task execution framework,” in Proceedings of the2007 ACM/IEEE conference on Supercomputing. ACM, 2007, p. 43.

[16] Z. Guo, A. Bhuiyan, A. Saifullah, N. Guan, and H. Xiong, “Energy-efficient multi-core scheduling for real-time dag tasks,” vol. 76, 2017.

[17] B. Keshanchi, A. Souri, and N. J. Navimipour, “An improved geneticalgorithm for task scheduling in the cloud environments using thepriority queues: formal verification, simulation, and statistical testing,”Journal of Systems and Software, vol. 124, pp. 1–21, 2017.

[18] M. I. Daoud and N. Kharma, “A high performance algorithm forstatic task scheduling in heterogeneous distributed computing systems,”Journal of Parallel and distributed computing, vol. 68, no. 4, pp. 399–409, 2008.

[19] G. C. Sih and E. A. Lee, “A compile-time scheduling heuristicfor interconnection-constrained heterogeneous processor architectures,”IEEE transactions on Parallel and Distributed systems, vol. 4, no. 2,pp. 175–187, 1993.

[20] Y. Huang, N. Bessis, P. Norrington, P. Kuonen, and B. Hirsbrunner,“Exploring decentralized dynamic scheduling for grids and cloudsusing the community-aware scheduling algorithm,” Future GenerationComputer Systems, vol. 29, no. 1, pp. 402–415, 2013.

[21] V. Hamscher, U. Schwiegelshohn, A. Streit, and R. Yahyapour, “Evalu-ation of job-scheduling strategies for grid computing,” in InternationalWorkshop on Grid Computing. Springer, 2000, pp. 191–202.

[22] R. Ranjan, M. Rahman, and R. Buyya, “A decentralized and cooperativeworkflow scheduling algorithm,” in 8th IEEE International Symposiumon Cluster Computing and the Grid, 2008. CCGRID’08. IEEE, 2008,pp. 1–8.