A Scalable Algorithm for Finding Delay-

Constraint Least-Cost End-to-End Path

Yue Han1,2, Zengji Liu1, Mingwu Yao1, and Jungang Yang1,2

1 Xidian University, ISN State Key Lab, Xi’an, China2 Xi’an Institute of Communications, Xi’an, China

Abstract. The Delay-Constrained Least-Cost(DCLC) routing problemis known to be NP complete, hence various heuristic methods have beenproposed for this problem. However, these heuristic methods have poorscalability as the network scale increases. In this paper we propose a newmethod based on Markov Decision Process (MDP) theory to address thescalability issue of the DCLC routing problem. The proposed algorithmcombines the benefit of the hierarchical routing with the advantage ofthe probabilistic routing in decreasing the advertisement of the networkstate information. Simulation results show that the proposed methodimproves the scalability significantly.

Keywords: DCLC, MDP, Scalability.

1 Introduction

In this paper, we focus on the most typical delay-sensitive routing problem whichhas been extensively studied in the past decade, the delay-constrained least cost(DCLC) routing problem [1], i.e., to find a path that has the minimal cost subjectto a delay constraint.The DCLC routing plays a very important role in the field ofQoS routing. Since it is a NP-complete problem, many heuristic algorithms havebeen proposed in the literature. The earliest works are Hassin’s two ε approxima-tion algorithms [1], which have very high time complexities. R.Widyono proposeda Constrained Bellman-Ford (CBF) algorithm [2] that performs a breadth-firstsearch to find the optimal DCLC path, but it is not practical for large scalenetworks due to the exponential time complexity with the increasing networksize.

To solve the DCLC problem in polynomial time, two types of combined pathheuristic algorithms are introduced. One is Delay Constrained Unicast Routing(DCUR) algorithm [3] and the other is Lagrange Relaxation based AggregateCost (LARAC) algorithm [4], [5]. In spite of the reasonable time complexity,these methods can not guarantee the optimality of the derived path. Recently,some new modifications have been made on the basis of DCUR and LARACin [6], [7] and [8]. Although these methods find near-optimum solutions withimproved time complexity, they do not always get the optimum path. In fact,the scalability problem is a long-standing problem for DCLC routing and so farthe study on the scalability DCLC problem is still an open issue.

Y. Tan, Y. Shi, and Z. Ji (Eds.): ICSI 2012, Part II, LNCS 7332, pp. 407–413, 2012.c© Springer-Verlag Berlin Heidelberg 2012

408 Y. Han et al.

Although existing algorithms have achieved low time complexity, they can notguarantee finding an optimal path even if it exists. In addition, these algorithmsstill have a significant scalability problem as the network size increases, which isthe fundamental problem of any QoS routing. For DCLC routing, the scalabilityis the ability of effectively deploying the algorithm at a larger network scaleoffering 100% success ratios in finding the optimal path. Hence, our task inthis paper is to present a theoretical method for computing the optimal DCLCrouting in which the scalability is improved.

The scalability problem of QoS routing in large scale networks can be solvedby using hierarchical routing strategies [9] and probabilistic routing [10]. So far,however, they have never been used in DCLC routing. The underlying idea of ourmethod is to formulate the DCLC routing as a Markov Decision Process(MDP)[11] based on the hierarchical routing and the probabilistic routing. The methodcan be used regardless of the network size and the performance remains stablesimultaneously. In addition, it always finds the optimal solution with identicalcomplexity for various network size. Thus the scalability problem is settled.

2 The Proposed Method for DCLC Routing

In this section, we present our MDP-based method for DCLC problem. Due tothe flat character of traditional MDP routing, we rebuild a two-level MDP modelin the multi-domain network for determining the inter-domain links and theintra-domain links respectively. Thus, we can combine the probabilistic routingwith the hierarchical routing to propose a new DCLC routing algorithm. SeeFig.1 for graphical illustration of the two-level MDP model.

Fig. 1. illustration of the two-level MDP model

The customarily used MDP models include the finite-horizon cost model andthe infinite-horizon discounted cost model. We formulate the upper level MDPas an infinite-horizon discounted cost MDP. The times of leaving each domainare defined as the decision time n. At each n, the congestion status of the inter-domain links is defined as the state in. in are assumed to be statistically inde-pendent. This assumption implies that a packet arrival carried on an n-link pathis decomposed into n independent packet arrivals at each single link. It is entirelyjustified as the connectionless property of DCLC routing. The link selection isdefined as the action λn. As a result of each action, a cost is expended according

A Scalable Algorithm 409

to a transition probability determined by λn and a new state in+1 at the nextdecision time n+1 is achieved. We define Pu(in+1|in, λn) as the transition prob-ability and Ru(in, λn) as the cost. Thus, the upper level MDP is expressed by a5-tuple {n, in, λn, P

u(in+1|in, λn), Ru(in, λn)}.

Depending on the action λn at in, the lower level MDP is determined. Weformulate it as a finite-horizon cost MDP. Its description is similar to the up-per level MDP. Thus, the corresponding 5-tuple of the lower level MDP isexpressed by {ti, xti , ati , P

l(xti+1 |xti , ati , in, λn), Rl(xti , ati , in, λn)}.We assume

a fixed time T after which the inter-domain link decision is made. That is,ti = (tnT , ..., t(n+1)T−1), where tnT = n + ε and ε is a positive number arbi-trarily close to zero. As this process evolves through ti, a sequence of cost isreceived according to a policy. We assume that the network is steady, so thatthe stationary policy can be used. We define the stationary policy πl for theintra-domain link selection. Let V l(xtnT , π

l) represent the T-horizon expectedtotal cost if πl is used and xtnT is the initial state. It is defined by (1), wherethe subscript in, λn on E signifies that in and λn are fixed for the expectation.

V l(xtnT , πl) =

t(n+1)T−1∑

ti=tnT

ExtnT

in,λn(Rl(xti , π

l, in, λn)

+ ExtnT

in,λnRl(xt(n+1)T

))

(1)

V l(xtnT , πl) may act as a single-step cost for the upper level MDP. As the selec-

tion process of inter-domain links evolves through n, a sequence of cost whichis the sum of V l(xtnT , π

l) and Ru(in, λn) is received. We define the stationarydecision policy πu for the inter-domain link selection. Let V (x, i, π) representthe infinite-horizon expected total discounted cost if a stationary decision pairπ = (πl

n, πun) is used and x, i is the initial state. It is defined by

V (x, i, π) =∞∑

n=0ξnEx,i(V l(xtnT , π

l) +Ru(in, πu))

=∞∑

n=0ξnEx,i(

t(n+1)T−1∑

ti=tnT

ExtnT

in,λn(Rl(xti , π

l, in, πu)

+ExtnT

in,λnRl(xt(n+1)T

)) +Ru(in, πu))

(2)

where ξ is the discount factor, which is a important parameter to characterizenetwork load. A lower ξ may be selected to indicate the lower network load,whereas a higher ξ may be selected to indicate the higher load. Our work is tofind the least cost path subject to a delay constraint, if such delay-constrainedpaths exist. The delay is then observed to find all the feasible paths. Meanwhile,V (x, i, π) may be mapped as either a monetary function or some weight of thenetwork throughput. Thus an optimal policy π∗ that minimizes V (x, i, π) is em-ployed for directing the link selection. As a stationary decision, the link selectiondecisions can be made locally at each node according to the optimal policy, thatis, the routing decision is independent of the network scale. Thus the scalabilityproblem is settled. With the MDP theory, the following result holds, which isrelated to the optimal V (x, i, π). Therefore, we omit the proof

410 Y. Han et al.

V ∗(x, i, π) = infπ

∞∑

n=0ξnEx,i(V l(xtnT , π

l) +Ru(in, πu))

= infπ

∞∑

n=0ξnEx,i

π R(in, λn)

= infπ

∑

λ

π0(λ|i){(R(i, λ)

+ξ∑

y

∑

j

PT (y|x, πl(i, λ))Pu(j|i, λ)V (y, j, π′)}

(3)

where R(in, λn) is the sum of V l(xtnT , πl) and Ru(in, λn), π

′ = (π1, π2...) andPT (πl(i, λ)) is the T steps transition probability following πl(i, λ). From theconclusion of MDP theory [11], there exists an unique solution v∗ that satisfies(3). Then the argument that achieves the unique solution v∗ is the optimal policyπ∗. The detailed steps for solving π∗ is referred to [12].

According to the MDP theory, the computational complexity of the proposedalgorithm is O(x ·a · i ·λ) and the communication complexity is O(x ·a), where xand i are the state space distribution of the intra-domain link and inter-domainlink respectively, a and λ are the action space distribution of the intra-domainlink and inter-domain link respectively. Whereas the worst case complexity ofDCUR is O(V 3), where V is the number of nodes, and the complexity of LARAChas the same order of complexity as Dijkstra’s algorithm. Since the computa-tional complexity and communication complexity of our method are independentof network scale, that is, the proposed method has a free choice of the number ofnodes/links in the network to be used. Hence it is especially suitable for DCLCrouting to achieve better scalability with the fast network expansion. A detailedexample for explaining the improvment on the scalability is referred to [12].

3 Performance Simulation

We evaluate the performance of the proposed method, Hierarchically ParallelTime Variation (HPTV) DCLC, by comparing it with LARAC and DCUR.The simulations run on OPNET Modeler. The tandem multi-domain network isused to provide larger network scale. The Waxman’s method [13] is employed togenerate each domain topology. For each link, the delay is uniformly distributedon specific interval[1ms, 10ms] and the cost is uniformly distributed from 0 to 1.The delay constraint is chosen on the specific interval[60ms, 100ms]. The statespace, action space and transition probability are determined as shown in Table1 of [12].

Fig.2 shows the average cost under different delay constraints. It indicatesthat the results of HPTV are relatively small compared to LARAC or DCURand get even better stability as the delay constraint varies. This is attributedto the fact that several available paths with low delay are always selected forHPTV to find the optimal path and thereby result in the improved performance.

Fig.3 shows the average cost under a specified delay constraint (75ms) for alarger network (20 traversed nodes). As the figure illustrates, HPTV outperformsLARAC or DCUR in terms of the average cost and the variation is practically

A Scalable Algorithm 411

Fig. 2. Average cost under different delay constraints

very small which, because of using the optimal policy, provides accurate pathselection. The results indicate the larger network does not introduce greater costsand hence account for better scalability when the network scale becomes larger.

Fig. 3. Average cost under 75ms constraint

Fig. 4. Average delay under various network sizes

The comparison of the average delay and average cost under various networksizes is shown in Fig.4 and Fig.5, respectively. It shows that the results of HPTVare almost less than those of LARAC or DCUR. In addition, they are insensitiveto the increased number of traversed nodes on the network path, while thoseof LARAC or DCUR vary significantly. This is because the unique stationary

412 Y. Han et al.

Fig. 5. Average cost under various network sizes

decision is selected on each traversed node for the repeated use, and the opti-mization is based on the entire network. This observation is consistent with theintuitive reasoning that these simulations verify the scalability of our proposemethod.

4 Conclusion

This paper proposes a scalable MDP-based routing algorithm for DCLC problem.To the best of our knowledge, our method is the first to apply MDP theory toDCLC routing. The algorithm can be used regardless of the network size andthe performance remains stable simultaneously. Thus the scalability is improved.Simulation results verify the validity of our proposed algorithm.

Acknowledgment. This work is supported by National Nature Science Foun-dation of China (NSFC) 61001129 and Nature Science Foundation of ShaanxiProvince 2011JM8033.

References

1. Hassin, R.: Approximation schemes for the restricted shortest path problem. Math-ematics of Operations Research 17(1), 36–42 (1992)

2. Widyono, R.: The design and evaluation of routing algorithms for realtime chan-nels. Technical Report TR-94-024, University of California at Berekley (1994)

3. Reeves, D.S., Salama, H.F., Fang, Y.: A distributed algorithm for delay constrainedunicast routing. IEEE/ACM Trans. Networking 8(20), 239–250 (2000)

4. Juttner, A., Szviatovszki, B.: Lagrange relaxation based method for the QoS rout-ing problem. In: INFOCOM, Alaska, pp. 239–250 (2001)

5. Korkmaz, T., Krunz, M.: Multi-constrained optimal path selection. In: INFOCOM,Alaska, pp. 834–843 (2001)

6. Sola, Y.E., Pourjavad, M.A.: Load balancing effects in DCUR QoS routing algo-rithm. In: ICCSIT, Beijing, pp. 2273–2278 (2009)

7. Agrawal, H., Grah, M.: Optimization of QoS Routing. In: ICIS, Melbourne, pp.598–603 (2007)

A Scalable Algorithm 413

8. Mohemmed, A.W., Sahoo, N.C.: Hybrid co-evolutionary particle swarm optimiza-tion and noising metaheuristics for the delay constrained least cost path problem.Journal of Heuristics 16(4), 593–616 (2010)

9. Dragos, S., Collier, M., Grah, M.: Macro-routing: a new hierarchical routing pro-tocol. In: GLOBECOM, pp. 1510–1514 (2004)

10. Ghosh, D., Acharya, R.: A probabilistic scheme for hierarchical QoS routing. In:ICON, pp. 416–421 (2001)

11. Puterman, M.L.: Markov decision processes: discrete stochastic dynamic program-ming. Wiley-Interscience, Mathematics (2005)

12. Han, Y., Liu, Z.-J.: A two-time scale MDP-based routing algorithm for DCLCproblem. In: IEEE ChinaCom, pp. 1–5 (2009)

13. Waxman, B.M.: Routing of multipoint connections. IEEE JSAC 6(9), 1617–1622(1988)