Datacenter InterconnectK. Christodoulopoulos, K. Kontodimas, A. Siokis, K. Yiannopoulos, and E. Varvarigos
Abstract—The NEPHELE data center interconnectionnetwork relies on hybrid electro-optical top-of-rackswitches to interconnect servers over multi-wavelengthoptical rings. The bandwidth of the rings is shared, andan efficient utilization of the infrastructure calls for co-ordination in the time, space, and wavelength domains.To this end, we present offline and incremental dynamic re-source assignment algorithms. The algorithms are suitablefor implementation in a software defined network controlplane, achieving efficient, collision-free, and on demandcapacity use. Our simulation results indicate that the pro-posed algorithms can achieve high utilization and lowlatency in a variety of traffic scenarios that include hotspots and/or rapidly changing traffic.
Index Terms—Dynamic resource allocation; Matrixdecomposition; Scheduling; Slotted and synchronous oper-ation; Time-wavelength-space division multiplexing.
I. INTRODUCTION
T he widespread availability of cloud applications to bil-lions of end users and the emergence of platform- and
infrastructure-as-a-service models rely on concentratedcomputing infrastructures, the data centers (DCs). DCstypically comprise of a large number of interconnectedservers running virtual machines. As traffic within theDC (east–west) is higher than incoming/outgoing traffic,and both are expected to continue to increase [1], DCnetworks (DCNs) play a crucial role. High throughput, scal-able, and energy/cost efficient DCN networks are requiredto fully harness DC potential.
State-of-the-art DCNs are based on electronic switchingin fat-tree topologies [2]. Fat-trees tend to underutilize re-sources, require a large number of cables, and suffer from
poor scalability and low energy efficiency [3,4]. To reducecost, fat-trees are typically oversubscribed (e.g., 1:4), anddo not offer full bisection bandwidth (FBB) that may beneeded for certain applications. Application-driven net-working [5,6], an emerging trend, would benefit from anetwork that flexibly allocates capacity where needed.
To cope with the shortcomings of fat-trees, many recentworks proposed hybrid electrical/optical DCN, a survey ofwhich is presented in Ref. [7]. The authors of Refs. [8,9] pro-posed a DCN in which heavy long-lived (elephant) flows areselectively routed over an optical circuit switched (OCS)network, while the rest of traffic goes through the elec-tronic packet switched (EPS) network. These solutions relyon the identification of elephant flows, which is rather dif-ficult, while it was observed that such long-lived heavyflows are not typical [4], making it difficult to sustain highOCS utilization. Instead, a high connectivity degree isneeded [4]. To enable higher connectivity, Ref. [10] pro-posed and prototyped a very dense hybrid DCN that alsosupports multi-hop connections, along with a custom builtcontrol stack. The authors measured the total delay, includ-ing control plane and OCS hardware reconfiguration (mi-croelectromechanical system—MEMS—switches), to be ofthe order of hundreds of milliseconds. Multi-hop routingwas exploited anew as shared optical circuits inRef. [11], where an OpenFlow (OF)-based control planewas developed [12], showing that circuit sharing reducesthe effect of slow OCS reconfigurations.
Other proposed DC interconnects completely lack electri-cal switches. Proteus, an all-optical DCN architecture basedon a combination of wavelength selective switches (WSSs)and MEMS was presented in Ref. [13]. Again, multi-hopis used to achieve high utilization. However, it is still hardto compensate the MEMS slow reconfiguration timesthrough sophisticated control. References [14,15] introducedhybrid OCS and optical packet/burst switching (OPS/OBS)architectures, controlled using SDN. Various other architec-tures based on OPS/OBS were proposed [7,16] (and referen-ces therein). However, OPS/OBS technologies are not yetmature, so the current target could be small-scale networkswith limited upgradability potential.
The authors of Ref. [17] presented a hybrid DCN architec-ture called Mordia, which uses WSS to provide switchingtimes of 11.5 μs. Mordia operates in a dynamic slotted man-ner to achieve high connectivity. However, the scalability ofhttps://doi.org/10.1364/JOCN.9.001145
Manuscript received June 26, 2017; revised October 6, 2017; acceptedOctober 22, 2017; published November 29, 2017 (Doc. ID 300915).
K. Christodoulopoulos (e-mail: [email protected]), K. Kontodimas,and E. Varvarigos were with the Department of Computer Engineeringand Informatics, University of Patras, Greece. They are now with theSchool of Electrical and Computer Engineering, National TechnicalUniversity of Athens, Greece.
A. Siokis is with the Department of Computer and Engineering andInformatics, University of Patras, Greece.
K. Yiannopoulos is with the Department of Informatics andTelecommunications, University of Peloponnese, Greece.
E. Varvarigos is also with the Department of Electrical and ComputerSystems Engineering, Monash University, Australia.
Mordia is limited as it uses a single wavelength divisionmultiplexing (WDM) ring whose capacity can accommodateonly a few racks, while resource allocation algorithmsexhibit high complexity and cannot scale to large DCs.
The European project NEPHELE is developing anoptical DCN that leverages hybrid electrical/opticalswitching with SDN control to overcome current datacen-ter limitations [18]. To enable dynamic and efficient shar-ing of optical resources and collision-free communication,NEPHELE operates in a synchronous slotted manner.Timeslots are used for rack-to-rack communication andare assigned dynamically, on a demand basis, so as toattain efficient utilization, leading to both energy and costsavings. Moreover, multiple wavelengths and opticalplanes are utilized to implement a scalable and highcapacity DC network infrastructure.
The NEPHELE network relies on WSSs, which are fasterthan the MEMS used in Refs. [8–11] and more mature thanthe OPS/OBS used in Refs. [14–16]. The fast switching times,along with the dynamic slotted operation, provide high andflexible connectivity. Compared toMordia [17], which also re-lies onWSSs,NEPHELE ismore scalable: it consists ofmulti-ple WDM rings, re-uses wavelengths, and utilizes cheappassiveroutingcomponentsandscalableschedulingschemes.The latter is the major contribution of this paper, whichpresents fast scheduling algorithms to meet NEPHELE dy-namic reconfiguration requirements.
Regarding resource allocation, scheduling problemssimilar to those addressed in this paper were studied in thepast for satellite and ATM switches [19–26]. Indeed, onecan view the entire NEPHELE multi-ring DCN as a largedistributed switch. The key difference of our work is thatwe consider huge network installations and dynamic time-divisionmultiplexing (TDM) operation; thus strict optimalityis not the objective, but we rather target low complexity.
We also encounter certain internal collision constraints thatare particular to the NEPHELE architecture (Section VI),and thuswe need to extend previous TDM algorithms appro-priately.Apart from[17],which considersdynamicTDMoper-ation, a somehowrelevantalgorithmicwork is [27],where theauthors present an integrated optical network-on-chip (NoC)basedonaring topologyandmicro-ringresonators (MRs).ThekeydifferencewiththeNEPHELEnetworkis thatMRstargetNoC and small networks, where propagation and controlplane delays are negligible. Thus, scheduling does not takeplace in periods, as in NEPHELE, but on a per slot basis asin electronic switches [28].
The remainder is organized as follows. In Section II,we describe the NEPHELE architecture. In Section III, wedescribe the dynamic resource allocation problem. InSection IV we provide a set of algorithms to solve it. InSection V we analyze the resource allocation constraintinduced by the NEPHELE architecture. In Section VI weevaluate the performance of the proposed algorithms.Finally, we provide our conclusions in Section VII.
II. NEPHELE ELECTRICAL/OPTICAL INTERCONNECT
NEPHELE is a hybrid electrical/optical DCN architec-ture, built out of POD and top-of-rack (TOR) switches.Figure 1 describes the NEPHELE DCN. The NEPHELDCN is divided into P pods1 of racks. A pod consists of IPOD switches and W TOR switches, interconnected asfollows: each TOR switch listens to a specific wavelength(thus, by design, the number of wavelengths equals thenumber W of racks in a pod) and has I ports. Each portis connected to a different one of the I POD switches. A rack
POD Switch 1...
AWG
W x 1 power combin
er
1x2 fast
switch
AWG
1 x 2 WSS
R x 1 power
combiner
...
... ...
coupler
coupler
...
...
1 x 2 WSS
...
1x2 fast
switch
Ring DROP
Ring DROP
Tx 1 (tunable)Rx 1
TOR (1,1)
Rx I......
Rx 1
TOR (W,1)
Tx I (tunable)Rx I...
POD Switch P
R
1
...
AWGW x 1 power combin
er1x2 fast
switch
AWG
Cyclic AWG W x R
1 x 2 WSS coupler
R x 1 power
combiner
Ring PASS
...
... ...
coupler
coupler
......
1 x 2 WSS couplerRing PASS
......
...
1x2 fast
switch
Ring DROP
Ring DROP
Rx 1
TOR (1, P)
Tx I (tunable)Rx I...
... Rx 1
TOR (W, P)
Tx I (tunable)Rx I...
...
...
Plane 1
R
1
...
AWG
1x2 fast
switch
AWG
Cyclic AWG W x R
1 x 2 WSS coupler
R x 1 power
combiner
...
... ...
coupler
coupler
... ...
1 x 2 WSS coupler
...
...
...
1x2 fast
switch
R
1
...
AWG
1x2 fast
switch
AWG
Cyclic AWG W x R
1 x 2 WSS coupler
R x 1 power
combiner
...
... ...
coupler
coupler
... ...
1 x 2 WSS coupler
...
...
...
1x2 fast
switch
...
Optical Plane 1
Optical Plane I
pod 1
...
...Servers (S ports)
...Servers (S ports)
...Servers (S ports)
...Servers (S ports)
Tx 1 (tunable)
Tx 1 (tunable)
Tx 1 (tunable)
Tx I (tunable)
Wx1 power
combiner
Wx1 power
combiner
POD (1,1)
POD (I,1)
POD (1,P)
POD (I,P)
pod P
Fig. 1. NEPHELE DCN architecture.
1The term “pod” refers to the cluster of racks, and “POD” to a NEPHELE podswitch.
1146 J. OPT. COMMUN. NETW./VOL. 9, NO. 12/DECEMBER 2017 Christodoulopoulos et al.
consists of S (computer, storage, or memory) servers. TheTOR is a hybrid electrical/optical switch and each of theS servers of the rack connects to it via a link. Thus, aTOR switch has S ports facing “south” to the servers.
POD switches are interconnected viaWDM rings to form“optical planes.” An optical plane consists of a single PODswitch per pod (for a total of P POD switches in the DCN)connected with R fiber rings. Each fiber ring carries WDMtraffic over W wavelengths (W, by design, equals the num-ber of racks in the pod), propagating in the same direction.There are I identical and independent/parallel (in thesense that traffic entering a plane stays in it until thedestination) optical planes. In total, there are I · P PODswitches, W · P TOR switches, and I ·R fiber rings.
We now explain how communication is performed in theNEPHELE DCN (Fig. 1). The key routing concept is thateach TOR switch listens to a specific wavelength (out ofW available), and wavelengths are re-used among pods.The NEPHELE TORs use tunable transmitters that aretuned according to the desired destination. Each TOR em-ploys virtual output queues (VOQs) per TOR destination(W · P VOQ per TOR) to avoid head-of-line blocking.
Traffic in the form of an optical signal originating froma port (plane) of a TOR switch enters a POD switch andis switched through a fast 1 × 2 space switch according tolocality: if the traffic is destined to a TOR in the pod(intra-pod), it remains within the POD switch; otherwise,it is routed to the rings and to the next POD switch. Localintra-pod traffic enters aW × 1 power combiner, located afterthe 1 × 2 space switch, and then a 1 ×W arrayed waveguidegrating (AWG). The AWG passively routes the traffic, de-pending on the used wavelength, to the desired destination.
Inter-pod traffic is routed via the fast 1 × 2 switch towarda W × R cyclic AWG (CAWG) followed by couplers thatcombine the CAWG outputs into the R fiber rings. TheW ×R CAWG has a passive routing functionality, with theincoming signal being routed to the output port (ring):
r � �ws �wd − 1�modR; (1)
where 1 ≤ ws ≤ W is the input port (the index of the sourcerack in the source pod, which equals its listening wave-length), 1 ≤ wd ≤ W is the wavelength that has to be usedto reach the specific destination (thus, also equal to the in-dex of the destination rack in the destination pod), and“mod” denotes the modulo operation. In the simple (notcyclic) 1 ×W AWG, the output depends only on the usedwavelength. So, the traffic enters the ring according tothe CAWG function, propagates in the same ring throughintermediate POD switches, and is dropped at the destina-tion pod. These routing decisions are applied by setting ap-propriately the wavelength selective switches (WSSs) inthe related POD switches. The WSSs can select whethertraffic passes through or drops on a per-fiber, per-wavelength, and per-slot basis. Thus, each intermediatePOD sets the corresponding WSS to the pass state, whileat the destination the related WSS is set to the drop state.The drop ports of all the WSSs—corresponding to all therings—are connected to a power combiner and a 1 ×W
AWG. So again, the traffic once dropped is passively routedto the desired TOR according to the wavelength used.
Following the above, wavelengths are statically assignedto racks, to simplify optical routing, and are re-used for ef-ficient operation. Conflicts on the WDM rings are avoidedin the time and space (plane) domains. Regarding the timedomain, NEPHELE operates in a synchronous slottedmanner that closely resembles the operation of a single(huge and distributed) time-division multiple access(TDMA) switch. In particular, NEPHELE maintains thetimeslot component of TDMA, but timeslots are not stati-cally assigned; instead, a central scheduler dynamicallyassigns them based on traffic needs, enabling efficientutilization of the resources. However, making schedulingdecisions on a per-timeslot basis is prohibitive, due to highcommunication and processing latency. Instead, it is bothmore efficient and less computationally demanding toperform resource allocation periodically, so that schedulingdecisions are made for periods of T timeslots; this approachfacilitates the aggregation and suppression of monitoringand control data and also absorbs traffic peaks.
From the control plane perspective, configurable compo-nents are the tunable transmitters (I per TOR switch), the1 × 2 optical switches (W per POD switch), and the WSSs(R per POD switch). The timeslot duration is lowerbounded by the slowest component, which is the WSS witha switching time of about 10 μs [17]. This is reserved as aguardband and the timeslot is taken to be 0.2 ms, so thatthe network exhibits 95% efficiency. The amount of datatransmitted during a timeslot equals the wavelengthtransmission rate times the timeslot duration (i.e.,0.2 ms × 10 Gbps � 2 Mbits) and will be referred to as adata unit (DU). This is also the switching granularity ofa NEPHELE DCN. A reference number for T is 80timeslots, corresponding to a period of 16 ms.
The existence of I parallel planes provides an additionaldomain, the space, to resolve conflicts: each timeslot of eachplane can be independently allocated. We will refer to atimeslot/plane combination as a generic (time)slot, imply-ing that the space and time domains are interchangeable inNEPHELE.
Variations of the above described baseline NEPHELEarchitecture include cases where each TOR does not listento a specific wavelength. One such variation will be given inSection V. Still, the NEPHELE routing function remainssimilar: the transmitter needs to select the appropriatewavelength, which is pre-calculated based on certainparameters (the source, the destination, the plane, etc.,as opposed to only the destination in the baseline architec-ture), while the WSSs are configured according to thatwavelength mapping.
Since a CAWG is used to route the W wavelengths on Rrings, we must have W ≥ R in order for the CAWG to beable to use all R egress ports. This is a system constraint.We can also derive the required conditions for achievingFBB assuming that the NEPHELE network is nonblocking(see Sections IVand V). We say that a DC interconnect hasFBB if for any bisection of the servers in two equal parti-tions, each server of one partition is able to communicate at
full rate with any server of the other partition. Since a TORsupports S servers, the number of PODs connected to aTOR must be at least I ≥ S, so that all servers of a TORcan communicate with servers outside their rack.Considering the whole network, there are P ·W · S serverports, whereas the overall capacity in the POD-to-POD net-work is I · R ·W. Thus, for FBB, we need to haveI ·R ·W ≥ P ·W · S �> I ·R ≥ P · S. Assuming I � S, theFBB requirement becomes R ≥ P. More flexibility isobtained by increasing the number of planes I. In thepresence of traffic locality, the FBB requirement can be re-laxed to support larger DCs. Table I presents target valuessatisfying the above constraints (including FBB) for a fullyfledged NEPHELE network using commodity off-the-shelf(COTS) equipment and a reference DC size (32K servers).
III. BANDWIDTH ALLOCATION IN NEPHELE
NEPHELE architecture exploits the SDN concept thatdecouples data and control planes through open interfaces,enabling programmability of the networking infrastruc-ture. NEPHELE utilizes an optical network with I opticalplanes,R fibers/plane andW wavelengths/fiber to intercon-nect the TOR switches in P pods. As discussed above, thenetwork operates in a slotted and synchronous manner.A key functionality of the NEPHELE SDN controller is thecoordination of the networking resources usage, includingthe timeslot/plane dimension. Thus, an important buildingblock of the SDN controller is the scheduling engine, whichallocates resources to communicating TOR pairs in acentralized, periodic, and on-demand manner.2
Recall that the number of racks per pod is equal to thenumber of wavelengths, and each rack listens to a specificwavelength. A TOR switch s is thus defined by a uniquepair s � �ps; ws�, where ps, 1 ≤ ps ≤ P, is the index of thepod it belongs to, and ws, 1 ≤ ws ≤ W, is the rack indexwithin the pod (ws is also the wavelength on which TORs receives data). It will sometimes be convenient to re-present the TOR switch by the scalar index s �ps · �W − 1� �ws instead of the pair representation�ps;ws�; as the mapping between the two representationsis one-to-one, we will use, with a slight abuse of notation,the same symbol s to stand for the TOR itself, the scalarindex, and the pair representing it.
We assume that a Data period consists of T timeslots,and we denote by Q�n� the queue matrix for period n.The queue matrix Q�n� is of size �W · P� × �W · P�, andelement Qsd�n� corresponds to the number of DUs thatare queued at the start of period n at source TOR s and
have as destination TOR d, with s � ps · �W − 1� �ws,d � pd · �W − 1� �wd, 1 ≤ ws;wd ≤ W, and 1 ≤ ps; pd ≤ P.That is, Qsd�n� is the number of DUs in VOQ �s; d� atthe start of period n. Since the scheduling problems ofthe different wavelengths are not independent, we willavoid breaking this matrix per wavelength.
Two operation modes are envisioned for the NEPHELEnetwork: (i) application-aware and (ii) feedback-based net-working. The former approach [5,6] assumes that applica-tions communicate to the NEPHELE SDN controller(or via the DC orchestrator) their topology and traffic re-quirements. In that case, the queue matrix is constructedfrom input from the applications. The latter, feedback-based, mode assumes that the central controller collects(monitors) data from the TOR queues [9] to build the queuematrix. We can also have a hybrid application-aware andfeedback-based network. In the following we focus on thefeedback-based approach, which is the hardest of thetwo from the control and scheduling viewpoint. The analy-sis and the proposed algorithms are applicable with minorchanges to application-aware and hybrid operation.
Recall that matrix Q�n� records the queue sizes at thestart of period n. We denote by A�n� the matrix of arrivalsat the queues during period n and by S�n� the schedulecalculated for period n. Element Qsd�n� denotes the DUsin the �s; d� queue at the start of period n, elementAsd�n� the DU arrivals during period n, and elementSsd�n� the DUs scheduled to be transferred from s to d dur-ing period n. We will describe in the next section the wayschedule S�n� is calculated.
Under feedback-based operation, the NEPHELE net-work operates in two parallel cycles:
1) data communication cycles of T timeslots (also referredto as a Data period), where the actual communicationbetween TORs takes place, and
2) resource allocation cycles of duration C (measured inData periods of T timeslots), where control informationis exchanged. If the duration of the resource allocationprocess is not fixed, C is an upper bound on it.
Resource allocation cycle n corresponds to Data period n,and computes the schedule S�n� to be used during thatperiod. Note, however, that the schedule is computed basedon information that was available C periods earlier thanthe Data period to which the resource allocation cycle cor-responds (and is applied). Thus, S�n� is a function ofQ�n − C�, i.e.,
S�n� � f �g�Q�n − C���; (2)
where Q̂�n� � g�Q�n − C�� is the function that creates theestimated queue matrix Q̂�n� from Q�n − C� upon whichthe schedule is calculated, and f is the scheduling algo-rithm. When C > 1 period (control delay larger than theData period), a new resource allocation cycle still startsevery Data period. So, there are C resource allocationcycles (or virtual control planes) running in parallel. For
TABLE IFULLY FLEDGED NEPHELE NETWORK PARAMETERS
Parameter W P S I
Value 80 20 20 20
2In the following, the terms “bandwidth allocation,” “resource allocation,”and “scheduling” will be used interchangeably.
1148 J. OPT. COMMUN. NETW./VOL. 9, NO. 12/DECEMBER 2017 Christodoulopoulos et al.
determining the schedule S�n� to be used during Dataperiod n:
a) the traffic matrix engine of the SDN controller collectsthe queue sizes from the TORs to build Q�n − C� andruns the queue estimation algorithm g to create theestimated queue matrix Q̂�n� � g�Q�n − C��,
b) the scheduling engine of the SDN controller runs algo-rithm f to calculate the schedule S�n� � f �Q̂�n��, and
c) the SDN controller communicates the scheduling out-put S�n� to the NEPHELE data plane devices (PODand TOR switches) to be used during Data period n.
Figure 2 shows the NEPHELE resource allocation anddata cycles (control and data plane, respectively). As dis-cussed, there is a delay between the two cycles: scheduleS�n� applied in Data cycle n is computed based on queuematrix Q�n − C�, since it takes C periods to compute andreach the data plane devices. The queue evolution isdescribed by
Q�n� 1� � Q�n� � A�n� − S�n�; (3)
where S�n� � f �g�Q�n − C���. The valueC does not affect theachievable throughput, as long as scheduling decisions areefficient (more on that later), but affects the traffic delay.The control plane delay C depends on many factors, onthe execution time of the scheduling algorithm, and the de-lay of the control protocol carrying information from TORsto the SDN controller (if monitoring is assumed) and fromthe SDN controller to the data plane devices. Both delaysdepend on the network size and the choice of the Dataperiod T.
For scheduling decisions to be efficient, the schedulingmatrix S�n�, computed based on an estimated queue ma-trix Q̂�n�, which in turn is calculated by Q�n − C�, shouldbe a “good” scheduling to be used during Data intervaln. This is true when Q̂�n� is a good approximation ofQ�n�. For slowly and medium changing traffic, we expectcalculations made for previous periods to be valid. Inestimating Q̂�n� from Q�n − C�, it is possible to also usestatistical predictions, filters, and other (notably applica-tion-aware) methods to improve performance. Moreover,it is possible for the scheduler to fill unallocated resourcesin S�n� by opportunistic transmissions, which can have col-lisions or be collision free (e.g., nodes agree to use slots inlexicographic order, mimicking static TDM, which underheavy load is efficient). Finally, the overall scheme is“self-correcting”: if some queues are not served for some
periods due to poor scheduling and their size grows dueto new arrivals, this will be communicated with some delayto the controller, and the queues will eventually be served.A study of the effect of the control plane delay and ways tomitigate it is part of our future plans.
In the following we will focus on the scheduling problemin the NEPHELE network. We start from the estimatedqueue matrix Q̂�n� and devise fast algorithms to calculatethe schedule S�n� [function f in Eq. (2)]. For reference wecan assume that we calculate the estimated queue matrix[function g in Eq. (2)] as Q̂�n� � A�n − C − 1� � Q̂�n − 1�−S�n − 1�, where we acknowledge that due to control planedelay C, the central scheduler has access to (delayed)arrival information A�n − C − 1� instead of A�n�. This corre-sponds to the case where the schedule S�n� calculated onQ̂�n� serves the arrived traffic A�n − C − 1�, plus a correc-tion equal to traffic not served in the previousperiod Q̂�n − 1� − S�n − 1�.
We now describe the form of the schedule S�n�. Thescheduling engine provides the TOR pairs that communi-cate during each timeslot and for each optical plane withinthe upcoming Data period. Note that wavelengths andrings are dependent resources; the selected wavelengthis determined by the destination, and the ring dependson the source and destination according to Eq. (1). Thus,in NEPHELE the allocated resources are the timeslotsand the optical planes (I · T in total), or the generic slots,as stated previously.
The scheduling algorithm takes the estimated queuematrix Q̂�n� and decomposes it (fully or, if not possible, par-tially) into a sum of I · T permutation matrices P�n; g�,g � 1;…; I · T, each corresponding to a generic slot. A per-mutation matrix is binary of size �W · P� × �W · P�; an entryPsd�n; g� equals “1” if a DU is to be transferred from TOR sto TOR d during the gth generic slot of period n, and “0”otherwise. In other words, Psd�n; g� identifies if one DUat the d-VOQ of TOR s will be transmitted during thegth generic slot of period n.
A permutation matrix determines a configuration of thenetwork for a specific generic slot. For the communicationto be contention free, the scheduling constraints SC1, SC2,and SC3 that are summarized in Table II should be satis-fied. In particular, the first two constraints, SC1 and SC2,ensure that each TOR transmits to and receives from atmost one TOR per generic slot. Constraints SC1 and SC2
Fig. 2. NEPHELE resource allocation and data cycles.
are relevant to all TDMA-like architectures and are readilyenforced by the decomposition process.
The third constraint, SC3, is related to the (not nonblock-ing character of the) architecture, and particularly, it is aresult of the usage of static routed CAWGs as opposed todynamically configured components. To better illustrateSC3, assume that a source TOR �ws; ps� communicates witha destination TOR �wd; pd�. This communication takesplace over the optical ring that is calculated fromEq. (1), and it occupies wavelength wd on the ring segmentbetween ps and pd. If another source TOR �ws; ps0 � withinthe aforementioned ring segment (i.e., ps < p0
s < pd) concur-rently communicates with destination TOR �wd; pd0 �, acollision will occur irrespective of the destination pod(pd0 ), since it occupies the same ring andwavelength. A sim-ilar contention will occur if the destination pod lies in theinitial ring segment (i.e., ps < p0
d < pd), irrespective of thesource pod. Note that SC3 is alleviated for R ≥ W, which,however, leads to underutilization of rings. Moreover, theeffect of the lack of the nonblocking property for the archi-tecture (when seen as a huge switch), or equivalently theexistence of SC3, is small, and will be discussed inSections V and VI.
The set P�n; g�, g � 1;2;…; I · T, of permutationmatricescomprise schedule S�n�, which records information for allgeneric slots of period n. The permutation matricesP�n; g� are stored as sparse matrices, each with W · Pentries. Similarly, S�n� is sparse with I · T ·W · P entries.
IV. SCHEDULING ALGORITHMS
Having described NEPHELE DCN operation, we nowproceed to present a set of NEPHELE scheduling algo-rithms. We assume that we start with the estimated queuematrix Q̂�n� and calculate the schedule S�n� [function f inEq. (2)]. To target both static and dynamic resource alloca-tion scenarios, we developed two classes of schedulingalgorithms: (i) offline and (ii) incremental. Offline algo-rithms, given in Subsection IV.A, take the estimated queuematrix Q̂�n� and compute schedule S�n� “from scratch.”Incremental algorithms, given in Subsection IV.C, usethe previous schedule S�n − 1� and update it based on traf-fic changes to obtain S�n�. Offline algorithms are bettersuited for semi-static traffic, take longer to execute, andachieve better utilization; incremental algorithms arefaster and can handle dynamic scenarios.
A. Offline Scheduling
As discussed above, offline scheduling decomposesthe matrix Q̂�n� into a set of permutation matricesS�n� � fP�n; g�g, g � 1; 2;…; I · T, without taking intoaccount the previous decomposition. We start by present-ing the optimal offline scheduling algorithm.
The decomposition of Q̂�n� can be performed in anoptimal manner following the well-known Hall’s theorem(an integer version of the Birkhoff–Von Neumann theorem
[24]). We define the critical sum [�HQ̂�n�� � h of matrixQ̂�n� as the maximum of its row sums and column sums.Then the following theorem holds:
Hall’s Theorem: An integer matrix of critical sum h canbe written as the sum of h permutation matrices.
The following algorithm calculates the optimal decompo-sition of matrix Q̂�n�:
1. Find a matrix of nonnegative integers E�n� so that ma-trixM�n� � Q̂�n� � E�n� is a perfect matrix with criticalsum H�M�n�� � H�Q̂�n�� � h. A perfect matrix has thesum of each row and each column equal to the criticalsum. An algorithm to obtain E�n� is found in Ref. [21].
2. Treat M�n� as a (bipartite) graph adjacency matrix andobtain a maximum matching j → p�j�, j � 1; 2;…; P ·W.This matching can then be represented as a permuta-tion matrix P�n; i�, whose �j; p�j�� entries are equal to 1,and all other entries are 0.
3. Find the weight ci as the smallest element of M�n� thatcorresponds to a nonzero entry in P�n; i�.
4. Repeat P�n; i� for ci times in the schedule andupdate M�n� � M�n� − ci · P�n; i�.
5. IfM�n� is not equal to zero, repeat steps 2–4. Otherwise,an optimal decomposition for M�n� has been found.
6. Set the entries of the dummy matrix E�n� to zero.
Steps 2–4 are repeated h times at most and we have thatPici � h. Note that the decomposition of an integer matrix
as a sum of h permutation matrices is not unique and thatthe permutation matrices in the decomposition ofM�n� arefull rank (corresponding to full utilization of the I · Tgeneric slots), while those in the decomposition of Q̂�n� �M�n� − E�n� may not be full rank [leaving some genericslots unused, namely, the entries of E�n�, and availablefor opportunistic transmissions]. In general, decomposi-tions that use a limited number of permutations, each car-rying a considerable amount of traffic ci, are preferable asthey result in fewer reconfigurations in the NEPHELEswitches.
The preceding algorithm assumes that the critical (rowor column) sum is constrained, but this will not always bethe case. The arrival matrix A�n� corresponds to traffic cre-ated by the servers and aggregated at the related TORswitches in period n. Since one link connects a server tothe TOR, the server sends to its TOR switch at most 1DU during a timeslot. Therefore, the row sums of A�n�are at most S · T. Some of A�n� ’s column sums, however,may be larger than that, e.g., in the presence of hotspot des-tinations. Note that the capacity connecting a TOR to thedestination servers can transfer S · T DU, and this is thesame for all DCNs. So hotspot problems, where traffictoward some TORs (columns of A) exceeds the availablecapacity, are present in all DCNs and not only inNEPHELE.
We could, in principle, devise flow control mechanismsto guarantee that the critical sum of A�n� satisfiesH�A�n�� ≤ S · T. Using an entry flow control mechanismbetween servers and source TORs, like the “stop and go”
1150 J. OPT. COMMUN. NETW./VOL. 9, NO. 12/DECEMBER 2017 Christodoulopoulos et al.
queueing proposed in Ref. [29], which limits (smoothens)the entry of DUs toward the destinations, we can enforcethe column sum to be less than S · T. In particular, eachsource TOR can check the destination TOR d of the pack-ets forwarded to it by the source servers and, through abackpressure mechanism, guarantee that packets equiva-lent to at most �S:T�∕�W:P� DUs are destined for each des-tination during a period of duration T. Such a source flowcontrol mechanism, however, may be too restrictive, unnec-essarily, and introduce large entry delays, as packets arequeued at the servers, outside the interconnection network.To relax somewhat the constraint, a credit-based flow con-trol mechanism can be used at the pod level, where eachsource POD is givenWd � �S:T�∕P credits for each destina-tion TOR d per period, which it can distribute to the TORsbelow it that can, in turn, distribute them to the servers.This would relax considerably the input flow control con-straints and the corresponding delays at the servers, butrequires a clever mechanism for distributing credits.
Even if a flow control mechanism is not present, thecolumn sums will be on average ≤S · T, assuming the desti-nations of packets are uniformly distributed on average.Actually, the critical sum will be ≤S · T not only on averagebut also with high probability, if the network operates at lessthan full load. Finally, note that TCP flow control smoothensthe traffic to a given destination. Since the downstream linksfrom a TOR to the servers can support up to S · T DU perData period, the previous condition will tend to hold withhigh probability in a DC network that employs TCP.
Based on the previous discussion we conclude that in the“typical case” the column sums of the arrival matrix A�n�will be ≤S · T and so will also be its critical sum (since therow sums are always ≤S · T). In that case, the scheduleS�n�, that is calculated based on Q̂�n� � A�n − C�, assum-ing S ≤ I, can be chosen so as to completely serve all thearrivals in A�n − C� in the available I · T generic slots.Note that in the reference FBB network scenario we as-sume S � I and so we will interchangeably use S and Iin the following. Thus, in this case, all packets generatedin a Data period will be served C periods later, emptyingthe queue from such packets. So the delay in theNEPHELLE network is upper bounded by C periods whenappropriate input flow control is used, or with high prob-ability when the load is far enough from full load. Thus, inthe typical case, NEPHELLE provides both full throughputand delay guarantees.
In themore general case where the critical sum of Q̂�n� isnot bounded by I · T, we stop when we find the first I · Tpermutations, while the traffic Q�n� − S�n� that is notserved is fed to produce the estimated matrix for nextperiod Q̂�n� 1�. Fairness and priority issues can also behandled with small extensions to the above process withouta requirement for additional flow control.
B. Complexity of Offline Scheduling and Stability
For general traffic, we define the load intensity betweensource destination TOR pair �s; d� as
ρsd�A� � E�Asd�∕�I · T�; (4)
whereE�� stands for expected value and 0 ≤ ρsd�A� ≤ 1 for aFBBNEPHELE network (S � I). The load intensity matrixP�A� is defined as the matrix with ρsd�A� entries. The rowsums of P�A� are always less than or equal to 1, while for astable network (finite queues), the column sums shouldalso be less than or equal to 1.
Necessary condition for stability: For the NEPHELEnetwork to be stable, the load intensity matrix P�A� shouldbe at most a double stochastic matrix.
When the previous condition does not hold, it is impos-sible to finda schedule to serve the queues ofNEPHELE ina stable manner. It is thus up to the DC orchestrator toallocate tasks to servers so that their communication re-quirements meet this constraint. Our target is to provideschedules that can serve any (long-term) stablematrix P�A�.
We define the average network load ρ�A� (alsorepresented by ρ) for arrival matrices A as the scalar
ρ�A�� ρ�Xsd
ρsd�A�∕�P ·W��Xsd
E�Asd�∕�I ·T ·P ·W�; (5)
and 0 ≤ ρ�A� ≤ 1. The quantity ρ · P ·W · I · T equals theaverage of the entries of arrival matrix A during a period(or, equivalently, ρ · P ·W · I is the average number ofarrivals per timeslot and TOR-to-TOR pair).
Besides the load, another parameter that is important incharacterizing the arrival process and the algorithmic com-plexity is the arrival matrix density δ�A�, which is comple-mentary to the sparsity of A. In particular, if we define theindicator function 1��, as 1�x� � 1, when x > 0 and 1,otherwise, then the density δ�A� of matrix A is defined as
δ�A� � E�X
sd
1�Asd��∕�W · P�2; (6)
where E�Psd1�Asd�� is the average number of nonzero en-tries of A and, clearly, 0 ≤ δ�A� ≤ 1. In other words, δ�A� isthe fraction of nonzero entries of A. Then, the number ofnonzero entries M�A� is given by M�A� � δ�A� · �W · P�2.
In the worst case, the optimal algorithm described ear-lier executes a maximum matching algorithm I · T times(uniform traffic). Finding a maximum matching can betime consuming, and even the well-known Hopcroft–Karp bipartite graph matching algorithm [25] exhibitscomplexity of O�M�A� ·
������������W · P
p�, whereM�A� is the number
of nonzero elements in A. The number of different matchesis ρ · I · T, and thus the complexity of the optimal offlinealgorithm is O�ρ · δ · I · T · �W · P�52�.
An indicative example of the execution time required foroptimal decomposition with the Birkhoff–Von NeumannandHopcroft–Karp algorithms is shown in Fig. 3, for a fullyfledged NEPHELE network (parameters listed in Table I).The algorithm was developed in MATLAB and the simula-tions were performed on an Intel Core i5 laptop. Figure 3plots the average execution time of the optimal decompo-sition algorithm against the load ρ and density δ, whichare shown to range from tens of seconds to minutes.
In these simulations, the traffic was created as follows: ateach period, each source TOR communicated with δ ·W · Puniformly chosen TOR destinations by transmitting atotal number of ρ · I · T DUs.
Based on the above result, and given the size of a fullyfledged FBB NEPHELE DCN (Table I), we deduce that,even with an optimized software and hardware environ-ment, the optimal algorithm would only be viable undera static resource allocation scenario, where traffic remainsunchanged for prolonged periods. The requirement fordynamic resource allocation can be pursued vianon-optimal algorithms that employ maximal rather thanmaximum matchings, at the expense of blocking at highloads. To this end, we also developed faster offline heuris-tics of reduced complexity and performance quite close tothe optimal. In particular, we developed a greedy offlinealgorithm of complexity O�ρ · �W · P�2 · I · T�, which is lin-ear in the size of the problem [note that the number ofDUs to be scheduled is O�ρ · �W · P�2� and the number ofresources is O�I · T�]. For brevity, we do not discuss thisalgorithm, as it still cannot meet dynamic resource allo-cation requirements, but describe a variation of it inthe next subsection. To further reduce scheduling com-plexity, we have to exploit the variations (temporal andspatial) of traffic, as is done in the incremental schedulingalgorithms of the next subsection.
C. Incremental Scheduling Algorithms for LocalityPersistent Traffic
Itisevidentfromthepreviousresultsthatofflineschedulingisnot suitable forbursty traffic.Measurements incommercialDCs indicate that application traffic can be relatively bursty,with flows activating/deactivating within milliseconds [4].Although traffic can be bursty, it tends to be highly locally per-sistent: a server tends to communicate with a set of destina-tions that are located in the same rack or the same cluster/pod [4]. This is due to the way applications are placed inDCs, each occupying only a small fraction of the DC.
TOR switches in NEPHELE aggregate the flows of theservers in a rack, smoothening out the burstiness of
individual flows, especially considering locality persistenttraffic. To formally define locality persistency, we define thearrival matrix difference as DA�n� � A�n� − A�n − 1�, theload ρ�jDA�n�j�, and the density δ�jDA�n�j� of the differenceby replacing A with jDAj in Eqs. (5) and (6), where j · jstands for the entrywise absolute value.
Locality Persistency Property: holds if
δ�jDA�n�j� ≪ 1: (7)
We also define the estimated queue matrix difference asDQ̂�n� � Q̂�n� − Q̂�n − 1�. Note that when arrivals have thelocality persistency property [i.e., Eq. (6) holds], then, inview of the Section III discussion, we also expectδ�jDQ̂�n��j� ≪ 1. For example, in the typical case whereQ̂�n� � A�n − C − 1� � Q̂�n − 1� − S�n − 1�, the persistencyproperty of A also holds for the estimated matrix Q̂.
Motivated from this observation, we propose and inves-tigate incremental scheduling, i.e., rely on the previousschedule to calculate the new one. The expected benefitis that we need to update only specific elements of the per-mutation matrices of the decomposition of Q̂�n� 1�, corre-sponding to traffic that has changed, but there is no need tomodify the rest of the elements.
To be more specific, let Q̂�n� be the estimated queuematrix and S�n� be the schedule produced at period n. Tocompute schedule S�n� 1� for the next period n� 1 withestimated queue matrix Q̂�n� 1�, we perform thefollowing:
1. Compute DQ̂�n� � Q̂�n� 1� − Q̂�n� [Fig. 4(b)].2. Split DQ̂�n� into D��n� and D−�n�, where
0,1 0,3 0,5 0,7 0,90
25
50
75
100
125
150
175
Density 25%Density 2.5%Density 0.5%
Network Load
Exe
cuti
on
Tim
e (s
)
Fig. 3. Average execution time of optimal decompositionalgorithm as a function of load ρ and arrival matrix density δ.
Fig. 4. Concept of incremental scheduling.
1152 J. OPT. COMMUN. NETW./VOL. 9, NO. 12/DECEMBER 2017 Christodoulopoulos et al.
• D��n� denotes the matrix consisting only of the positiveentries of difference matrix DQ̂�n�, and
• D−�n� denotes the matrix consisting only of the negativeentries of difference matrix DQ̂�n�.
3. Use algorithm A1 to free entries of S�n� according tomatrix D−�n� and obtain the half-filled schedule, de-noted as S0�n� 1� [Fig. 4(c)].
4. Use algorithm A2 to add entries in S0�n� 1� (half-filledschedule) according to D��n� to obtain the currentperiod’s schedule S�n� [Fig. 4(d)].
Algorithm 1: Linear Greedy AlgorithmGiven: D��n�, S0�n� 1�, TC�n�, RC�n�, P, W, T, IReturn S�n� 1�, TC�n� 1�, RC�n� 1�1: S�n� 1� � fP�n� 1; g�g ← S0�n� 1�;2: TC�n� 1� ← TC�n�, RC�n� 1� ← RC�n�;3: for s ← 1 to P ·W do4: for d ← 1 to P ·W5: slots ← D�
sd�n�;6: g ← 1;7: while g ≤ T · I and slots > 08: if TCs�n� 1; g� � 0 and RCd�n� 1; g� � 0 then9: Psd�n� 1; g� � 1;10: TCs�n� 1; g� � 1; RCd�n� 1; g� � 1;11: slots ← slots − 1;12: end if13: g ← g � 1;14: end while15: end for16: end for
The complexity of incremental scheduling isO�δ�jDQ̂j� · ρ · I · T · �W · P�2�, where δ�jDQ̂j� ≪ 1 in view ofthe persistency property of Eq. (6) and the relateddiscussion.
The above describes the core of the incremental algo-rithms. In the first two algorithms that we will present,we used a greedy algorithm A1 in Step 3 to free entries thatworks as follows: by iterating each element of D−�n�, wefind the last permutation matrix of S�n� that serves thatelement, and we free that entry (set it to zero). This algo-rithm frees sequentially the scheduled resources for thedemands whose traffic was reduced, leaving the entriesthat satisfy the current traffic. Regarding step 3, wepresent threeA2 schemes, each corresponding to a differentincremental scheduling algorithm.
1) Linear-Time Greedy Incremental Heuristic: Thegreedy heuristic is a non-optimal algorithm running inlinear time to the size of the problem and the number ofgeneric slots per period. The greedy heuristic can be usedas an offline or as an incremental algorithm. In the follow-ing we focus on the incremental case. The algorithm takesas input the difference traffic matrix DQ̂(n) [or Q̂�n� 1� inoffline]. It follows steps 1–3 described above, so that it findsthe half-filled schedule, denoted as S0�n� 1� and the pos-itive differencematrixD��n�. By iterating on each non-zeroelement of D��n�, it greedily finds the available genericslots to use. This is done by taking into account constraintsSC1 and SC2, of Table II, which ensure that at each generic
slot a TOR can send to or receive from only one other TOR.Data structures TC�n� and RC�n� are used to keep track ofthese two constraints. In particular, element TCs�n; g� [orRCd�n; g�] records whether the transmitter (or receiver) atsource s (or destination d, respectively) and generic slot g isactive or not. The pseudo-code of the incremental greedyalgorithm is given in Algorithm 1.
2) Sublinear Greedy Incremental Heuristic: The sublin-ear greedy algorithm is a variation of the linear greedyheuristic, but it schedules blocks of DUs instead of DUs.In particular, an integer k � O�I� is chosen and used to cal-culate the block estimated queuematrix Q̂k�n� � Q̂�n�
k (in ourimplementation we chose k � 5, and I was a multiple of 5).The purpose of this procedure is to reduce the amount ofload to be scheduled, within a span of T · Ik generic slots,speeding up the scheduling process roughly by a factorof k. The block estimated queue matrix is treated as theestimated queue matrix, while applying the previousgreedy algorithm. The schedule produced by the greedy al-gorithm is reproduced k times, in order to cover the initialtraffic. As expected, the speedup obtained comes at a cost:dummy DUs are introduced when the ceiling function isapplied, which are allocated some generic slots, reducingthe resource usage. In particular the load overheadintroduced is
Number of dummy DUs �Xs;d
�k ·
�Q̂s;d�n�
k
�− Q̂s;d�n�
�:
(8)
In order for the algorithm to run in sublinear time(a speedup of roughly k is expected), some filtering hasto be applied to Q̂�n� in such a way that its critical sumis at most T·I
k after the division, rather than T · I. This proc-ess takes place in the estimated queue matrix creationmodule and requires at least linear time to complete.These two operations, however, namely, the estimatedqueue matrix creation and the scheduling, are performedby different modules. The queue matrix creation modulecan start executing while receiving monitoring informa-tion; once the block estimated queue matrix is created,the scheduling algorithm is executed in sublinear time.We consider this to be technically feasible for theNEPHELE’s architecture.
3) Randomized Heuristic: A randomized variation of thegreedy heuristic was also implemented for an incrementalresource assignment. Randomized operation avoids thegreedy first find approach, aiming to increase (on average)the traffic that is served [30]. The algorithm follows an ap-proach similar to the four steps presented at the start ofthis subsection: it receives as input the previous period’sschedule S�n�, the estimated queue matrix Q̂�n� 1�, andcalculates the schedule S�n� 1�. In the first phase, itexamines the previous period’s permutation matricesP�n; g� against the traffic they can carry in the newperiod and discards any P�n; g� that carries less trafficthan
PsdQ̂sd�n� 1�∕�I · T�, expecting that a new random-
ized allocation could provide a better solution for the
corresponding generic slot. The P�n; g� that carry their fairshare of the traffic load are then subtracted from Q̂�n� 1�:
1. If the subtraction of a P�n; g� leaves no negative entries,then the P�n; g� is kept unaltered in S0�n� 1�.
2. Whenever negative entries occur, the correspondingentries on both P�n; g� and Q̂�n� 1� are set to zero,and the updated P�n; g� is used in S0�n� 1�.
The previous steps calculate (i) the updated set of per-mutations S0�n� 1�, by skipping the calculation of D−�n�,and (ii) the positive change matrix D��n�, which is theQ̂�n� 1� matrix after the subtractions. In this case,D��n� includes the new connections, the old connectionswith increased traffic, and the old connections thatbelonged to discarded permutations. Then the entries ofD��n� are distributed randomly on S0�n� 1� followingthe algorithm below:
1. Select a random destination TOR (column) d of D��n�.2. Find the m active source TORs for destination d, corre-
sponding to rows fs1; s2;…; smg of the non-zero entriesin column d, and re-arrange them randomly.
3. For each row sk in the randomized arrangement:a. Find the existing P�n� 1; g� that are available
for the �sk; d� communication (by checking therelated scheduling constraints—using the datastructures TC�n� and RC�n�, as discussed inSubsection IV.C.1).
b. If the number of available P�n� 1; g� is greater thanthe D�
sk;d�n� entry (i.e., more resources are available
than those required), randomly select the requirednumber; otherwise select all of them.
4. Repeat steps 1–3 for all columns of D��n�.
Finally, if any traffic remained in D��n� and not all theI · T permutations are utilized, then the algorithmperforms a final round where it repeats steps 1–4, withthe only difference being that new permutations areconsidered to be initially available to all connections.
V. ARCHITECTURE-RELATED CONSTRAINT
The resource allocation problem at hand is quite similarto scheduling problems for TDM satellite or ATM crossbarswitches [19–23]. Scheduling constraints SC1 and SC2 arecommon, but constraint SC3 (Table II) is new and is a resultof specific architecture choices, and particularly of usingstatic routed (C)AWGs instead of reconfigurable compo-nents. This design choice, which was decided to keep thecost and routing complexity low, results in a NEPHELEDCN (when seen as a huge switch connecting TORs) losingits nonblocking character even for I � S. In the previoussection, we described algorithms that operate withouttaking into account SC3, whose effect is studied here.
To evaluate the performance under the additional con-straint SC3, we extended the incremental greedy heuristicSubsection IV.C.1) to account for SC3. The algorithm to bedescribed is referred to as the ring-segment greedy
heuristic. To be more specific, consider a transmission fromsource TOR s � �ws; ps� to destination TOR d � �wd; pd� atgeneric slot g (timeslot t over optical plane i), where ps < pd
without loss of generality. Such a communication is repre-sented in the schedule by Psd�n; g� � 1. Under the baselinearchitecture of Section II that uses W ×R CAWGs at theinput of the rings, this communication uses wavelengthwd and ring rsd � ��ws �wd − 1�modR�, according toEq. (1). So, the communication from s to d captures thering-wavelength resource, indexed
lsd � ��ws �wd − 1�modR� ·W �wd: (9)
Resource lsd is actually captured only for the segmentof the ring that is between pods ps and pd and can be usedby other connections if they use non-overlapping segmentsof the ring. SC3 constrains that s to d communication can-not take place simultaneously with communication froms0 � �ws0; ps0� to d0 � �wd0; pd0�, with ps < ps0 < pd or ps <pd0 < pd and wd0 � wd and �ws0 −ws�modR � 0 (seeTable II).
The ring-segment greedy heuristic algorithm keepstrack of the utilization of the ring-wavelength resourcesand the specific ring segments utilized. To accommodatethe communication from s to d at generic slot g, we needto check whether ring-wavelength resource lsd is used be-tween pods ps and pd. If it is not used, we reserve it to blockany future conflicting communication. The data structurerecords for each generic slot g � 1; 2;…; I · T the ring-wavelength resource l � 1; 2;…; R ·W and the specific ringsegment it uses (P ring segments in the worst case), result-ing in size O�P ·R ·W · I · T�. This data structure can besimilar to TC�n� and RC�n� used to keep track of SC1and SC2 (Subsection IV.C.1), which, however, are of sizeO�P ·W · I · T�. Specifically, line 8 of the pseudo-code ofAlgorithm 1, should also search for maximum P ringsegments, which increases the complexity.
The worst case traffic pattern is obtained when we havethe maximum number of conflicting communication pairsdefined by SC3, and all of them carry maximum traffic.Regarding the constraint on the overlapping of ring seg-ments, there are P such conflicting �s; d� pairs for unidirec-tional traffic (p1 to pP, p2 to p1;…; pP−1 to pP; pP to pP−1),and since they are in different pods they can have maxi-mum traffic equal to Q̂sd�n� � S · T. In this case, we requireI � P · S planes to fully serve the worst case traffic. Suchworst case traffic is, of course, highly improbable to occur.Still, our simulations show that the throughput is affectedeven in the average case when considering SC3, whilethe execution time increases, since we need to accountfor the ring segment utilization.
We developed two solutions to address this problem:the first extends the incremental greedy algorithm ofSubsection IV.C.1, considering in a more coarse way theutilization of the ring-wavelength resources, while thesecond relies on a variation of the architecture that usesspectrum-shifted optical planes.
1154 J. OPT. COMMUN. NETW./VOL. 9, NO. 12/DECEMBER 2017 Christodoulopoulos et al.
A. Full-Ring Greedy Heuristic
In the first solution, called the full-ring greedy heuristicalgorithm, communication from s to d is taken to occupythe entire ring-wavelength resource lsd, i.e., the whole ringand not only the segment between pods ps and pd. Thisreduces the size of the data structure needed to O�R ·W ·I · T� and improves the execution time over the ring-segment greedy heuristic discussed above, sacrificingsomewhat the (already lower) throughput performance.
B. Spectrum-Shifted Optical Planes
A problem of the baseline NEPHELE architecture isthat, if two communicating source–destination pairs,�s; d� and �s0; d0�, conflict over an optical plane, by usingthe same ring-wavelength resource lsd � ls0d0 � l, they willuse the same resource l and conflict over all planes. Thisproblem affects all planes, so we have available only thetime domain (T) to resolve such conflicts, as opposed tohaving both the plane and time dimensions (all I · T genericslots), resulting in lower performance. To address this, wedeveloped an architecture variation where the opticalplanes are spectrum shifted. To be more specific, in the ar-chitecture of Fig. 1, traffic for destination TOR d � �wd; pd�always uses wavelength wd. The main idea of spectrum-shifted optical planes is to make the ring-wavelength inEq. (9) depend on plane i and on other source/destinationlocation parameters. This proposed variation uses thedesired passive components, i.e., (C)AWGs, instead ofreplacing them by reconfigurable ones that wouldsignificantly increase the cost, due to their high radix.
The goal is to design all-pair conflict-free optical planes,so that TOR pairs conflicting on some optical plane do notconflict on another one. There are various ways to achievethat, such as permuting the rings between pods, or varyingthe CAWG routing function by changing the way CAWGsare coupled/added to the rings. One such efficient solutionis to replace the 1 ×W AWG connected to the drop ports ofthe WSSs with an P ×W CAWG connected as follows: Weconnect the drop ports of all the WSSs of plane i and pod pthrough the R × 1 power combiner to input port z�i; p�, 1 ≤z�i; p� ≤ P of the P ×W CAWG. The W output ports of theP ×W CAWG are connected to the TORs as before. Wemake the wavelength wsd�i�, used for communication be-tween source s � �ws; ps� and destination d � �wd; pd� overplane i depend on s, d, and i, as opposed to the baselinearchitecture where this was fixed and equal to wd.Considering the routing function of the CAWG, wsd�i�should satisfy the following condition in order to reachthe desired destination:
�wsd�i� � z�i; pd� − 1�modW � wd; (10)
where wd in this equation indicates only the location of thedestination TOR in the related pod (and not, as previously,the receiving wavelength), and z�i; pd� is the input portof the CAWG. The routing function of the CAWG that addsthe traffic to the rings at the source gives the ring used:
rsd�i� � �ws �wsd�i� − 1�modR: (11)
Then, the ring-wavelenth resource of plane i that is usedis
lsd�i� � ��ws �wsd�i� − 1�modR� ·W �wsd�i�: (12)
Consider now another TOR pair communication s0 ��ws0; ps0� → d0 � �wd0; pd0� on the same plane i. To create con-flict, this communication has to use the same wavelengthand the same ring with the s → d communication, i.e.,
Our goal is to avoid pairs s → d and s0 → d0 to conflict inany other plane. This can be satisfied if jz�i; pd�−z�i; pd0 �j ≠ jz�i0; pd� − z�i0; pd0 �j, for all 1 ≤ i0 < I, i0 ≠ i.Generally, we want that to hold for any conflicting pairof any plane, i.e., we need the following to hold for alli; �i0i0 ≠ i�, all pd; pd0 :
Remember that 1 ≤ z�i; p� ≤ P, since z�i; p� correspondsto the input port of the P ×W CAWG that the WSSs ofpod p at plane i are connected. For a prime number of podsP, one choice (along with others) that satisfies Eq. (14) is
z�i; p� � �1� �p − 1� · �i − 1��modP: (16)
For prime P, with the above function we construct Pall-pair conflict-free planes. The number of planes Irequired to serve any pattern is then I ≥ P. To see this, as-sume that we have several conficting pairs on a plane (P isthe maximum number of pairs, as discussed previously),and each requires the full capacity (all the timeslots) ofthe plane. This plane can serve any of those, but the re-maining pairs conflicting on that plane are not conflictingon the other I − 1 planes. Thus, if I ≥ P (which also holds forthe reference NEPHELE architecture—Table I), conflictscan be solved using the plane dimension in addition tothe timeslot dimension. In that case, the entireNEPHELE network is actually a nonblocking time–wavelength–space switch.
If P is not prime (in the reference P � 20), the abovefunction constructs all-pair conflict-free planes equal tothe smallest divisor (= 2 for the reference architecture).However, even in this case, the conflicts are reducedsubstantially. The average performance improves whenthe number of conflicting pairs among the planes is small,and the proposed solution reduces this number. All-pairconflict-free planes mean that this number is zero, whichresults in the best worst case and average performance.
We rely on simulations to evaluate the performance of oursolution for average traffic.
The extensions needed in the scheduling algorithm toaccount for spectrum-shifted planes are straightforward,and require the calculation of the wavelength based onthe source, destination, and plane. This can be done withpre-calculated tables and does not affect the complexity. Wealso need to use either the ring-segment or the full-ringheuristic algorithm to keep track of ring-wavelength re-source utilization. We decided to use the faster full-ringgreedy heuristic in the performance evaluation section.
VI. PERFORMANCE EVALUATION
A. Evaluation Without Architecture Constraint SC3
The proposed incremental scheduling algorithmswere evaluated via simulations for various trafficscenarios. We assumed a NEPHELE network withW � 80 racks∕pod, P � 20 pods, S � 20 server ports/rack,and I � S � 20 optical planes (see Table I), and set T � 80timeslots. We used a custom traffic matrix generator wherewe could control the following parameters [31]:
1. the average network load ρ�A�, defined from Eq. (4) asthe ratio of the total traffic over the total capacity. Theindividual TOR loads ρsd�A�were generated as indepen-dent Gaussian random variables, assuming that a TORaggregates a large number of TCP/UDP flows. Thedistribution mean was set equal to the desired load,while its variance was correlated to the load dynamicityρ�jDAj�;
2. the load dynamicity ρ�jDAj�, defined as the averagechange in traffic between successive periods;
3. the connection density δ�A�, defined from Eq. (6). Lowconnection density corresponds to a small number ofdestinations per source, thus an increased number oftraffic hotspots. To accommodate the description of traf-fic patterns of previous works [4], where TORs system-atically prefer to communicate with peers in specificpods, or even the same pod, we further distinguishedbetween intra-POD density δin�A� and inter-PODdensity δout�A�. A locality parameter is then definedas the traffic percentage that is destined within thesame pod over the total load:
l � δin ·Wδin ·W � δout ·W · �P − 1� ;
given that the local POD comprisesW TORs out ofW · Pthat are available in total; and
4. the locality dynamicity δ�jDAj�, defined as the averagenumber of connections that change from active toinactive and vice versa at each period. Traffic exhibitslocality persistency [Eq. (7)] when δ�jDAj� is low.
To evaluate the proposed algorithms, we developeda simulator in MATLAB. For each simulation instance,we chose to vary one parameter, while the rest of the
parameters were set to their default values (Table III).To focus on the performance of the scheduling algorithms,we assumed a resource cycle with C � 1, which corre-sponds to the schedule being calculated within a Dataperiod. We also assumed the reference case where the es-timated queue matrix on which the schedule is calculatedbased on the arrivals: Q̂�n� 1� � A�n − C� � Q̂�n� − S�n�.As discussed in Subsection IV.C, this ensures that the per-sistency property of A is also true for the estimated queuematrix Q̂.
For each parameter set, we measured a) the additionalaverage queuing latency, i.e., the average number of peri-ods a packet remains buffered in addition to the C � 1period that it takes for the schedule to be calculated, soas to focus on the efficiency of the algorithm and not ofthe whole control cycle, and b) the schedulimg algorithm’sexecution time (s) against the network load. We also mea-sured the maximum network throughput, defined as themaximum offered load at which the queues and the latencyare finite. Thus, the maximum throughput indicates theload that can be transferred by the network under stableoperation. Note that maximum throughput is identifiedin the latency/load graphs as the load at which the latencybecomes (assymptotically) infinite.
1) Queuing Latency: Initially, we present the results onthe latency. In the first set of simulations, the examinedparameter is intra-POD density, which is set to 100% forthe results of Fig. 5(a) and to 2.5% for Fig. 5(b); the otherparameters were set to their default values (Table III).Figure 5(a) shows that the sublinear greedy heuristicclearly underperforms, as expected, the other two algo-rithms, resulting in average latency that increases at load0.7 and becomes (asymptotically) infinite at load 0.8(= maximum network throughput). The linear greedy heu-ristic comes next, followed by the randomized heuristicwith slightly better performance. In the results of the sec-ond set of simulations, shown in Fig. 5(b), the density ofintra-POD connections is set very low to 2.5%. The queuinglatency of all three algorithms start to increase at loadaround 0.7. The increase is steeper for the sublinear greedyheuristic, followed by the linear greedy, and finally by therandomized heuristic. The latter two algorithms have verysimilar performance regarding latency and stability.
Locality considerably impacts the performance. Thegreedy and the random algorithms are more efficient whenthe matrix is concentrated in small blocks [δin high, heavyintra-POD traffic—Fig. 5(a)] than when it is spread out[Fig. 5(b)]. In contast, the sublinear algorithm underper-forms when locality is high; introducing several dummy
1156 J. OPT. COMMUN. NETW./VOL. 9, NO. 12/DECEMBER 2017 Christodoulopoulos et al.
DUs in a small block increases the column sum more thanwhen traffic and the locations of the dummy DUs arespread out.
We next examine the effect of the locality dynamicityparameter δ�jDAj�. When δ�jDAj� � 0.1% [Fig. 6(a)], allthree heuristic algorithms start to induce high latency atnetwork load of about 0.7. As in the previous cases, thequeuing latency increase with network load is steeperfor the case of the sublinear heuristic, followed by the lin-ear greedy, and then by the randomized heuristic. This ismore clear at load 0.8, where the sublinear greedy heuristicis already in the unstable region, while the linear greedyand the randomized heuristic remain stable until load 0.85.
When the locality dynamicity parameter δ�jDAj� � 10%[Fig. 6(b)], all three algorithms improve their results by in-creasing their maximum throughput (latency asymptotemoves to the right). Higher dynamicity reduces the persist-ency of bad scheduling matrices, improving the perfor-mance, but as expected, has negative effects onexecution times, as will be discussed in the following.
2) Scheduling Algorithms Execution Times: Next, wepresent results on the execution times of the consideredalgorithms. We provide four plots for the same parametersexamined in Subsection VI.A.1.
As shown in Fig. 7(a), the algorithms’ performance in or-der of increasing execution times is randomized, linear
greedy, and sublinear greedy heuristic. As expected, theaverage execution times increase with the load. At load0.8, the randomized heuristic needs an average of 1.5 sto complete. Next comes the linear greedy heuristic withan execution time (at 0.8 load) of about 0.7 s, and last comesthe sublinear greedy heuristic with about 0.5 s. These re-sults were expected from the theoretical complexity analy-sis given in Section IV. The relative order of the algorithmswith respect to their execution times remains thesame when intra-POD connection density is set to 2.5%[Fig. 7(b)]. The decrease in the execution times for lowintra-POD density is due to the fewer connections, eachof higher load, which reduces the complexity of all threealgorithms. The execution times for different values oflocality dynamicity parameter δ�jDAj� are depicted inFig. 8. As expected, by complexity analysis, execution timeincreases as load and locality dynamicity δ�jDAj� increases.
3) Maximum Network Throughput: We now focus on themaximum network throughput achieved by the schedulingalgorithms, defined as the load at which the queues and thelatency become (asymptotically) infinite and the system be-comes unstable. The throughput is examined with respectto two parameters that were not discussed above: (i) theinter-POD connection density δout and (ii) the load dynam-icity ρ�jDAj�. The results are shown in Table IV. We see thatthe impact of inter-POD connection density δout is quite sig-nificant, since for dense traffic (δout � 50%), the throughput
Fig. 5. Average queuing latency resulting from the examinedscheduling algorithms, measured in Data periods additional tothe control cycle, for intra-pod density δin equal to (a) 100% (locality68%) and (b) 2.5% (locality 5%).
Fig. 6. Average queuing latency resulting from the examinedscheduling algorithms, measured in Data periods additional tothe control cycle, for locality dynamicity δ�jDAj� equal to(a) 0.1% and (b) 10%.
reaches about 0.97, while for sparse traffic, it drops to0.85 at most. The reason is similar to the one discussedfor the role of intra-POD density. It should be noted that,for dense inter-POD connections (δout � 0.5%), the sublin-ear greedy heuristic is unstable even at low traffic loads,since it wastes too much capacity. This should be expected,as small and spread demands result in many entries thatcreate many dummy DUs, thus wasting network capacity.Regarding load dynamicity, we consider the cases ρ�jDAj� �0.1% and ρ�jDAj� � 10%. We observe that this parameterdoes not affect substantially the throughput, nor the exe-cution time. The throughput performance of all thealgorithms was similar, with the sublinear greedy heuristicbeing slightly worse and faster (lower than 0.4 s in almostall cases).
B. Evaluating the Effect of the SC3 Constraint
We evaluated the performance of the NEPHELE net-work under the architecture constraint SC3 and also forthe architecture variation that uses the spectrum-shiftedplanes. In particular, we assessed the performance for
In all examined cases, the number of planes was thesame (I � 20). Case (a) was examined in the previous sub-sections and is used here as a reference. The network ofcase (a) can achieve maximum throughput; that is, it canaccommodate any traffic if an optimal algorithm is used.The network of cases (b) and (c) has worst-case traffic thatrequires more (20 times) planes, while case (d) also re-quires more planes than the I available, but lower thanthose of cases (b) and (c). The probability of generatingthe worst-case traffic is extremely low, but cases (b) and(c) have several traffic instances that require more thanI planes, while for case (d) this probability is low. Note,however, that we use a heuristic (incremental greedy)and thus blocking is expected even for case (a).
Figure 9(a) shows the latency for density between podsδout � 50%, corresponding to l � 2.5% locality (defaultδin � 25%). Such a low locality results in heavy utilization
Fig. 7. Execution times of the algorithms for intra-pod density δinequal to (a) 100% (locality l � 68%) and (b) 2.5% (locality l � 5%).
Fig. 8. Execution times of the algorithms considered for localitydynamicity δ�jDAj� equal to (a) 0.1% and (b) 10%.
TABLE IVMAXIMUM THROUGHPUT OF ALGORITHMS CONSIDERED AS A
FUNCTION OF INTER-POD CONNECTION DENSITY δout AND LOAD
DYNAMICITY ρ�jDAj�Parameter Symbol Value Linear Randomized Sublinear
1158 J. OPT. COMMUN. NETW./VOL. 9, NO. 12/DECEMBER 2017 Christodoulopoulos et al.
of the inter-pod WDM rings and creates SC3 conflicts. Weobserve that the asymptotic throughput of the referencearchitecture/segment-ring greedy reduces to 0.8 comparedto 0.9 of the reference architecture/greedy, where SC3 is ne-glected. The reference architecture/full-ring greedy haseven lower throughput, measured to be 0.7, but exhibitslower execution times (see the following). The spectrum-shifted planes architecture resolves conflicts in one planeby serving in another plane and thus improves thethroughput. The achieved throughput was 0.85, which isclose to the case where SC3 is neglected, as shown by thereference architecture/greedy (no SC3).
As locality increases, inter-pod traffic decreases, and even-tually, at high locality, the performance of all algorithmsconverges. For example, in Fig. 9(c), where the density be-tween pods is δout � 0.5% (or l � 70% locality), we observethat the reference architecture/greedy (no SC3) achievesthroughput close to 0.95, very close to the rest of the casesexamined. Note that, according to [4], locality is very highin a Facebook DC, higher than 50% for typical DC applica-tions, such as web and map-reduce. Figure 10 shows the re-lated execution times. We observe that the referencearchitecture/segment-ring greedy has the highest runningtime, well above 1 s. Keeping track of ring segments yieldshigher complexity. Execution time is reduced in the referencearchitecture/full-ring greedy (but it wastes resources—haslower throughput, as seen in Fig. 9). The spectrum-shiftedplanes/full-ring greedy case has quite low execution time,similar to the reference architecture/full-ring greedy. Thus,it combines the execution time benefits of the full-ring
algorithm while achieving throughput close to the casewithout SC3 (by reducing the conflicting sets). As locality in-creases, the execution times of the reference architecture/full-ring and spectrum-shifted planes/full-ring converge tothat of the reference architecture/greedy (no SC3).
VII. CONCLUSIONS
Weproposed and evaluated a set of scheduling algorithmsfor the NEPHELE DCN. In NEPHELE, resources aredynamically allocated based on traffic requirements. Toavoid contention, a centralized allocation process enforcesthree scheduling constraints. We described in detail theNEPHELE control cycle, outlined its requirements, and pre-sented an algorithm to optimally allocate resources. We alsoproposed three incremental heuristic scheduling algorithmsthat reduce the execution times of allocation, and evaluatedtheir performance through simulations. The randomizedand greedy heuristics exhibited normalized throughputhigher than 0.85 for all examined traffic scenarios. The ex-ecution time of the greedy heuristic was measured in hun-dreds of milliseconds, while the sublinear greedy heuristicwas faster, sacrificing some throughput. The parallel imple-mentations of the proposed algorithms on specialized hard-ware (field-programmable gate array) to further reduceexecution time is ongoing. We also studied the effect on per-formance of the third scheduling constraint (SC3), which isspecific to the NEPHELE architecture. To cope with the re-sulting reduction of throughput and increase of execution
(a)
(b)
Fig. 9. Latency (in periods) as a function of load for density be-tween pods (a) δout � 50% (l � 2.5%) and (b) δout � 0.5% (l � 70%).
(a)
(b)
Fig. 10. Execution time as a function of load for density betweenpods (a) δout � 50% (l � 2.5%) and (b) δout � 0.5% (l � 70%).
time, we proposed an architecture variation that employsspectrum-shifted optical planes and extended the greedyheuristic to function in such a network. Simulations showedthat the throughput and execution time performance ap-proaches that of a network without SC3. The proposed incre-mental heuristic algorithms achieve high throughput andlow execution time, asserting the dynamic and efficientoperation of NEPHELE.
ACKNOWLEDGMENT
This work was partially funded by the EC through theHorizon NEPHELE project under grant agreementno. 645212.
REFERENCES
[1] “Cisco Global Cloud Index: Forecast and Methodology,2014–2019,” Cisco White Paper.
[2] M. Al-Fares, A. Loukissas, and A. Vahdat, “A scalable,commodity data center network architecture,” in ACMSIGCOMM, 2008, pp. 63–74.
[3] T. Benson, A. Akella, and D. Maltz, “Network traffic charac-teristics of data centers in the wild,” in ACM SIGCOMM,2010, pp. 267–280.
[4] A. Roy, H. Zeng, J. Bagga, G. Porter, and A. Snoeren, “Insidethe social network’s (datacenter) network,” in ACMSIGCOMM, 2015, pp. 123–137.
[5] J. Follows and D. Straeten, Application Driven Networking:Concepts and Architecture for Policy-Based Systems, IBMCorporation, 1999.
[6] M. Jarschel, F. Wamser, T. Hohn, T. Zinner, and P. Tran-Gia,“SDN-based application-aware networking on the example ofYouTube video streaming,” in IEEE European Workshop onSoftware Defined Networks, 2013, pp. 87–92.
[7] C. Kachris and I. Tomkos, “A survey on optical interconnectsfor data centers,” IEEE Commun. Surv. Tutorials, vol. 14,no. 4, pp. 1021–1036, 2012.
[8] N. Farrington, G. Porter, S. Radhakrishnan, H. H. Bazzaz, V.Subramanya, Y. Fainman, G. Papen, and A. Vahdat, “Helios:A hybrid electrical/optical switch architecture for modulardata centers,” in ACM SIGCOMM, 2010, pp. 339–350.
[9] G. Wang, D. G. Andersen, M. Kaminsky, K. Papagiannaki,T. S. E. Ng, M. Kozuch, and M. Ryan, “c-Through: Part-time optics in data centers,” in ACM SIGCOMM, 2010,pp. 327–338.
[10] K. Christodoulopoulos, D. Lugones, K. Katrinis, M. Ruffini,and D. O’Mahony, “Performance evaluation of a hybridoptical/electrical interconnect,” J. Opt. Commun. Netw., vol. 7,pp. 193–204, 2015.
[11] Y. Ben-Itzhak, C. Caba, L. Schour, and S. Vargaftik, “C-share:Optical circuits sharing for software-defined data-centers,”arXiv:1609.04521, 2016.
[12] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L.Peterson, J. Rexford, S. Shenker, and J. Turner, “OpenFlow:Enabling innovation in campus networks,” ACM Comput.Commun. Rev., vol. 38, no. 2, pp. 69–74, 2008.
[13] A. Singla, A. Singh, K. Ramachandran, L. Xu, and Y. Zhang,“Proteus: A topology malleable data center network,” in ACMSIGCOMM Workshop on Hot Topics in Networks, 2010,paper 8.
[14] S. Peng, B. Guo, C. Jackson, R. Nejabati, F. Agraz, S. Spadaro,G. Bernini, N. Ciulli, and D. Simeonidou, “Multi-tenantsoftware-defined hybrid optical switched data centre,”J. Lightwave Technol., vol. 33, pp. 3224–3233, 2015.
[15] G. Saridis, S. Peng, Y. Yan, A. Aguado, B. Guo, M. Arslan,C. Jackson, W. Miao, N. Calabretta, F. Agraz, S. Spadaro,G. Bernini, N. Ciulli, G. Zervas, R. Nejabati, and D.Simeonidou, “LIGHTNESS: A function-virtualizable soft-ware defined data center network with all-optical circuit/packet switching,” J. Lightwave Technol., vol. 34, pp. 1618–1627, 2016.
[16] N. Calabretta and W. Miao, “Optical switching in data cen-ters: Architectures based on optical packet/burst switching,”in Optical Switching in Next Generation Data Centers,Springer, 2017, pp. 45–69.
[17] G. Porter, R. Strong, N. Farrington, A. Forencich, C.-S. Pang,T. Rosing, Y. Fainman, G. Papen, and A. Vahdat, “Integratingmicrosecond circuit switching into the data center,” in ACMSIGCOMM, 2013, pp. 447–458.
[18] http://www.nepheleproject.eu/.[19] T. Inukai, “An efficient SS/TDMA time slot assignment algo-
rithm,” IEEE Trans. Commun., vol. 27, no. 10, pp. 1449–1455,1979.
[20] K. L. Yeung, “Efficient time slot assignment algorithms forTDM hierarchical and nonhierarchical switching systems,”IEEE Trans. Commun., vol. 49, no. 2, pp. 351–359, 2001.
[21] G. Bongiovanni, D. Coppersmith, and C. Wong, “An optimumtime slot assignment algorithm for an SS/TDMA system withvariable number of transponders,” IEEE Trans. Commun.,vol. 29, no. 5, pp. 721–726, 1981.
[22] D. Serpanos and P. Antoniadis, “FIRM: A class of distributedscheduling algorithms for high-speed ATM switches withmultiple input queues,” in IEEE INFOCOM, 2000.
[23] K. Y. Eng and A. S. Acampora, “Fundamental conditions gov-erning TDM switching assignments in terrestrial and satel-lite networks,” IEEE Trans. Commun., vol. 35, pp. 755–761,1987.
[24] G. Birkhoff, “Tres observaciones sobre el algebra lineal,”Univ. Nac. Tucuman Rev. Ser. A, vol. 5, pp. 147–151, 1946.
[25] J. Hopcroft and R. Karp, “An n5/2 algorithm for maximummatchings in bipartite graphs,” SIAM J. Comput., vol. 2,pp. 225–231, 1973.
[26] T. E. Anderson, S. S. Owicki, J. B. Saxe, and C. P. Thacker,“High-speed switch scheduling for local-area networks,”ACM Trans. Comput. Syst., vol. 11, no. 4, pp. 319–352,1993.
[27] I. Cerutti, N. Andriolli, P. Pintus, S. Faralli, F. Gambini,O. Liboiron-Ladouceur, and P. Castoldi, “Fast schedulingbased on iterative parallel wavelength matching for a multi-wavelength ring network-on-chip,” in Int. Conf. on OpticalNetwork Design and Modeling (ONDM), 2015.
[28] N. McKeown, “The iSLIP scheduling algorithm for input-queued switches,” IEEE/ACM Trans. Netw., vol. 7,pp. 188–201, 1999.
[29] S. Golestani, “A framing strategy for congestion manage-ment,” IEEE J. Sel. Areas Commun., vol. 9, no. 7,pp. 1064–1077, 1991.
[30] L. Tassiulas, “Linear complexity algorithms for maximumthroughput in radio networks and input queued switches,”in IEEE INFOCOM, 1998, pp. 533–539.
