Ethernet IEEE 802.3 [1] is the most successful Local Area Networking (LAN) technology.In 1998 Ethernet evolved to 1 Gbit/s over fiber (GbE) [2]. GbE maintains the simplicityand the frame structure of previous lower speed IEEE 802.3 standards but, as a matter offact, it definitely adopts a full duplex point-to-point configuration without the CSMA/CDprotocol. GbE spans maximum distances of many kilometers— more than 100 for the latest1000baseZX implementations over single-mode dispersion-shifted (DS) fiber. In June 2002the Optical Ethernet (OE) family was extended by approving the 802.3ae standard, 10 GbitEthernet (10 GbE) [3]. 10 GbE runs only over fiber in full-duplex mode, and it maintainsthe same frame format and size of IEEE 802.3. It defines two families of physical layer(PHY): (1) LAN PHY operating at a data rate of 10 Gbit/s, and (2) WAN PHY running ata data rate compatible with SONET OC-192c. Initially, 10 GbE implementations are beingdeveloped using existing SONET/SDH infrastructures for the transport of Ethernet frames(WAN PHY), thus extending the span of an Ethernet network across different countries.
However, the target is to implement transport networks based just on Ethernet directlyover optical layer, namely OE, possibly eliminating the SONET/SDH core infrastructure.The availability of low-cost OE interfaces, particularly on Layer 2 switches, has thereforeextended the use of Ethernet outside the LAN. These low-cost interfaces are emerging as anappealing solution for eliminating the bandwidth bottleneck between high-speed LAN andoptical metropolitan area network (MAN) core through an optical access network (OAN).
However the limiting factor for the full deployment of OE is represented by the lack ofcertain efficient operation, administration and maintenance (OA&M) features, such as faultdetection and recovery [4, 5]. Resilient schemes directly implemented at network Layer 2or on several upper layers, e.g., IP/MPLS resilient schemes, are currently available, but allof them exhibit some limitations.
Rapid Spanning Tree Protocol (RSTP [6]) acts at Layer 2. It is the evolution of IEEE802.1d Spanning Tree Protocol (STP) and is designed to reduce the time to recover networkconnectivity after an outage. The classical STP implementation detects failures using akeep-alive mechanism based on hello packets and recovers network connectivity withinone minute. RSTP improves this mechanism, and its recovery time is between 10 ms anda few hundred milliseconds. RSTP is already implemented in most OE switches, but therecovery time (which is sometimes higher than common SONET at 50 ms) represents anissue for some communications carriers.
Resilient Packet Ring (RPR) technology [7], is designed for metro ring topologies; itintroduces fairness and spatial reuse. It also supports three classes of services. The recoveryscheme of RPR exploits the possibility offered by a double counterrotating ring configura-tion of redirecting the packets away from the fault into the opposite ring direction (steering).Fault detection is based on a keep-alive mechanism, and connectivity recovery is obtainedwithin 50 ms. However, RPR requires specific topology (two counterrotating fiber rings)and a new MAC protocol that is not supported by most network devices.
Another typical metro access solution consists of several Layer 2 components posi-tioned at the metro access network edge and connected in a star topology to a Layer 3router, which is connected to the network core. Protection is achieved with a backup routerand different dedicated point-to-point OE connections between every switch, thus provid-ing protection against both core equipment and fiber failures. This solution provides a highthroughput infrastructure, but it requires the availability of many equipment ports. There-fore the initial cost of setting up the metro access infrastructure is high because the numberof ports needed in the device doubles. On one hand the former drawback does not heavilyimpact an implementation with only Layer 2 devices, because the cost of Layer 2 OE inter-faces is continuously decreasing. On the other hand, when customer premises need Layer3 equipment to connect a large number of users, e.g., as with companies, the increasednumber of edge router ports heavily impacts the overall setup cost.
Mesh networks consisting of IP routers connected by point-to-point OE links might im-plement additional resilient schemes at the IP/MPLS layer. MPLS fast reroute [8] achievesgood performance, but it typically requires Layer 3 equipment, the availability of sparealternative paths with other IP routers, and additional ports on each router.
The aforementioned solutions can be applied in various regions of an optical network.However, none of them represents a unique low-cost solution for all the possible networktopologies. Indeed in each solution there is a trade-off between resilience capabilities, pro-tocols complexity, and required network topologies.
Furthermore, though some solutions might appear redundant for a specific failure, theyare able to overcome additional failures, such as transponder or equipment failures. Forexample, solutions requiring equipment port duplication might also overcome transponderfailures.
In this paper we focus on providing the experimental demonstration of low-cost re-
silience solutions that are particularly suitable for arbitrary-topology OANs and based onthe OE architecture. Because the solutions are implemented at the optical layer, they aretransparent to different upper-layer resilience schemes, and they are not strictly related to aspecific network topology. The proposed solutions aim at minimizing the cost of guarantee-ing full recovery from a specific failure event: fiber link failure. The implementations differin the failure detection mechanisms, in the utilized restoration path activation, and in thenetwork scope they cover (from span protection to optical path protection). By using spaceswitch and/or photonic cross connect (PXC) driven by an out-of-band signaling through aLinux Box (LB), the proposed implementations do not require the duplication of the Layer3 IP router data ports.
The two proposed solutions are a low-cost 1:1 optical span protection particularly suit-able for point-to-point OE resilience and a shared OE path protection based on GMPLScontrol plane. Their performances are experimentally assessed, and results show that re-covery times in the millisecond range can be achieved. In addition, although the implemen-tation is carried out with consideration for a particular network architecture, the proposedsolutions are conceptually valid for any IP over WDM architecture regardless of the IPframing protocol used.
2. Optical 1:1 Dedicated Span Protection
The proposed low-cost implementations of the optical 1:1 span protection scheme are allbased on two IP routers, two Linux boxes (LB) and four 2× 1 opto-mechanical switches.The IP routers, equipped with GbE 1000BaseLX interfaces, are connected through theoptical switches (characterized by 6− 7 ms of switching time) to a pair (TX and RX) ofworking fibers, as shown in Fig. 1.
Customer Premises
GE
FE FE
MAN Core Point-to-Point Optical Access Network
Router A
Router
B
working
protection
Linux Box Linux Box
Fig. 1. Span protection setup.
When a link failure occurs, a detection mechanism notifies the LB, which in turn sendsthrough the parallel port a single bit to the 2× 1 switch, triggering the rerouting of theconnection to the protection fiber. As a result, the two routers are immediately switched tothe protection fibers. If, in the meantime, the working fibers are repaired, a failure of theprotection fibers will result in the reutilization of the working fibers.
Three failure detection mechanisms are proposed. The first one is based on the configu-ration shown in Fig. 1. Every LB runs a client application that continuously queries throughthe Fast Ethernet management port, which is a monitor daemon running on the router. Thedaemon is queried about the operation state (up or down) of the router GbE interface. Thesecond detection mechanism also can be implemented using the setup of Fig. 1 , and it isbased on a Simple Network Management Protocol (SNMP) agent running on the IP router.By using the Fast Ethernet management port every router notifies the LB if the operational
state of its GbE interface becomes down. Upon working fiber failure, the two routers inde-pendently change the state of their GbE interfaces from up to down. This event triggers theLB to force the use, through the switches, of the backup fibers. The third failure detectionmechanism implies a slight modification of the setup depicted in Fig. 1 , and it is shown inFig. 2.
Vref
Pin
Control Circuit
Linux Box
Router A
Fig. 2. Loss of Light (LOL) failure-detection mechanism.
It operates exclusively at the hardware level (Loss of Light) without interacting withthe IP router. The optical signal arriving at every router is partially spilled into a secondfiber so that the received signal, besides reaching the destination router, is also sent to acontrol circuit. The control circuit consists of a photodiode that converts the optical powerlevel in a voltage amplitude, which is compared to a threshold. When the level is below thethreshold, the LB receives through the parallel port this information that triggers the toggleof the switches.
3. GMPLS Shared Path Protection
The cost-effective implementation of fast Generalized Multiprotocol Label Switching(GMPLS) shared path protection in an IP over 10 GbE network presented in this sectionstems from the span protection scheme presented in Section 2. In the proposed GMPLSShared Path Protection scheme the LBs are part of an out-of band GMPLS control planeand they require proper definition of the GMPLS protocols for failure detection and traf-fic rerouting. In the data plane, the implementation exploits the introduction of intelligentlow-cost photonic cross connects (PXCs), i.e., all-optical switches that are driven by theGMPLS out-of-band signaling [9]. To limit the number of additional fibers to be used, asingle spare fiber is shared to protect connections spanning fiber disjoint paths between10 GbE router interfaces.
The testbed in which the GMPLS shared protection is implemented is shown in Fig.3. Two pairs of IP routers (A–B and C–D) equipped with 10 GbE interfaces are connectedthrough two working paths using physically disjoint fiber links. Transparent PXCs are uti-lized to share a single protection fiber (i.e., PXC3-PXC4 fiber). In the testbed implemen-tation, the utilized PXCs are low-cost, 4× 4 opto-mechanical switches characterized by atypical switching time of 3 ms. LBs are employed to manage the PXCs via the parallel port.GMPLS-based control messages, i.e., signaling protocol (RSVP-TE) and link managementprotocol (LMP) packets, are exchanged among the LBs on an out-of-band Ethernet con-trol plane. The protocol software is coded in C programming language, and it is based onstandard Libpcap and Libnet libraries. Each PXC is equipped with a control circuit (Fig.2) that provides the failure detection mechanism on the basis of Loss of Light (LOL), as
in the 1:1 dedicated span protection implementation. For example, as depicted in Fig. 3,when a failure occurs on the working fiber between PXC1 and PXC2, LB2 receives thefault detection notification. Then LB2 starts the LMP fault localization procedure by ex-changing LMP Channel Status messages with its adjacent LB1 (upstream in term of dataflow) through the control plane. LB1 then localizes the failure by checking the status of itsworking connection. At this point, the fiber PXC1-PXC2 is identified as the failure cause.LB1 initiates the fault recovery by sending an RSVP Path message to LB3, which controlsthe first downstream PXC on the precomputed backup path. As a result, the protection pathactivation request reaches the admission control module running on LB3. If the requestedresource (fiber PXC3-PXC4) is either available or, according to the priority level, it can bepreempted, LB3 sets the PXC3 for connecting PXC1 to PXC4, and it propagates the RSVPPath message to its downstream LB4. LB4 then continues the protection path activationfollowing the same procedure: it configures PXC4 to set up the connection to PXC2, andit propagates the message to LB2. The procedure ends when LB2 triggers the toggle of itscontrolled PXC2. As a result, the data traffic connection between routers A and B is trans-parently rerouted through the backup fiber. According to signaling protocol specifications[10], LB2 confirms the protection path activation by sending an RSVP Resv message alongthe reverse path until LB1 is reached.
Fig. 3. Shared protection implementation for 10 GbE.
4. Experimental Results and Discussion
4.A. Dedicated Span Protection
The proposed implementations for the dedicated span protection have been tested exten-sively using commercial routers. Working fibers have been interrupted more 100 timesfor each proposed solution. The resilience schemes always succeeded in the detection andrecovery of the failure. However, different recovery times have been experienced for thethree solutions. For each solution, the time required for the control PC to elaborate the faultreported by the detection mechanisms and to send the right command to the switches isnegligible (only a few microseconds).
In the first solution, operating on the scheme shown in Fig. 1, the LB receives a setof extensible markup language (XML) tags every 40 ms (lowest available refresh interval)from the daemon running on the router under test. This set of XML tags describes thestatus of the router interfaces. Because the router, by default, verifies and updates the stateof its interfaces once per second, the overall time necessary for the router to become aware
whether the link is faulty and to propagate this information to the control LB is large.Indeed a uniform distribution in the range of 80˘1080 ms has been observed.
The second software detection mechanism, implemented by an SNMP agent in the setupshown in Fig. 1, achieves similar results: the router evaluates the state of its interfaces onceper second and after a few tens of milliseconds it reacts by activating the protection system.In Fig. 4, the sample trace is shown. The computed average outage time is of almost 700 ms.
700 ms
Fig. 4. Typical outage time using the software failure-detection solutions.
Besides the large recovery time, the former two methods, based on querying the routerinterface status, have the drawback of introducing a routing table update. When a routerdetects that an interface is no longer active, it deletes all the destinations reachable throughthat interface from the routing table. Therefore, once the physical connectivity has beenrecovered, it is necessary to wait until the Open Shortest Path First (OSPF) routing entriesare restored to resume full operation of the router. On the other hand, since this solution isbased on a query to a router, it can be easily extended to handle more-complex recoveryschemes.
In the third solution, employing the LOL sensor (see Fig. 2) when the working fiberis affected by a failure, the communication is immediately switched to the back-up fibersusing the LBs as an intermediary. Because this solution is based on a protocol-agnosticcarrier sensing operation the speed of this solution is limited only by the switching timeof the 2× 1 switches, which is approximately 7 ms, as shown in Fig. 5. This detectionmechanism outperforms the previous two solutions in terms of recovery speed, and as apositive side-effect, the loss of OSPF entries in the routing table is very unlikely; in theperformed experiments it never occurred.
Each of the proposed solutions can be either singularly implemented or the three so-lutions can be combined to achieve high reliability. In case the third proposed solutiondoes not successfully recover the connection, the slower second one can intervene. In caseof failure of the second proposed solution, the first recovery mechanism can be triggered.The first recovery mechanism can also be used to verify the successful termination of therecovery procedure. The presented solutions do not detect signal degradation such as ex-cessive bit error rate (BER). However, the lack of this functionality does not represent animportant limitation because in OANs fiber cut is the most important cause of failure. Theproposed solutions represent viable alternatives for the implementation of low-cost reliableOANs in which customer sites are independently connected to the Metro access points.Indeed customer sites, if the proposed solutions are utilized, do not need to agree upon
Fig. 5. Outage time using the hardware failure-detection solution.
specific topologies (e.g., access ring) to be guaranteed resilience. Moreover, by protectingthe point-to-point access connections directly at the optical layer, the proposed solutionsavoid the duplication of expensive IP router interfaces.
4.B. Shared Path Protection
The proposed GMPLS implementation has also been extensively tested: the working fiberbetween PXC1 and PXC2 has been interrupted more than 100 times. The GMPLS sharedpath protection implementation always succeeded in the failure detection and recovery. Thetime required for LBs to elaborate the fault reported by the detection mechanisms is negli-gible (only a few microseconds). A few milliseconds are needed by the LMP localizationprocedure to isolate the fault and by the RSVP recovery process to complete the reroutingmessage exchange. Figure 6 shows the capture made by a sniffer application monitoring theLMP ChannelStatus and RSVP Path messages exchanged among the four involved LBs. Asshown in Fig. 6 the overall fault isolation and rerouting process takes less than 1.2 ms inthe Ethernet control plane that was used. Although this control plane represents an idealsituation, it has been shown [11] that running the control plane on a typical digital com-munication network (DCN) does not affect the fault localization and rerouting time. Froma data traffic perspective, the recovery procedure is concluded after the last PXC, namely,PXC2, is switched.
Fig. 6. GMPLS message exchange over the out-of-band control plane.
Figure 7 shows that the overall outage time for the destination Router B is less than
5 ms. Indeed the loss of about 40 packets is measured by a traffic analyzer when the datarate is 10 packets/ms. Since the router verifies, by default, the state of its interfaces onceper second, the interface down state is very unlikely: in the performed experiments it neveroccurred, and routers were never aware of the failure.
Fig. 7. Outage time (ms) at Router B.
The proposed implementation does not consider failures resulting from BER degrada-tion; however, especially in the urban context, fiber cut can be considered as the prevalentcause of failure. Although a particular network architecture is considered, the proposed so-lution is conceptually valid for any IP over WDM architecture regardless of the framingprotocol used. In addition, the solution can be applied in a differentiated reliability envi-ronment, where different levels of reliability can be offered at different prices, or it can beintegrated with other resilience schemes
5. Conclusions
We have presented and experimentally validated an optical 1:1 span protection techniquefor the OE using three failure detection techniques and fast protection activation. The so-lutions differ in recovery speed and implementation complexity. Separately or jointly im-plemented, they are suitable for being integrated into the control plane of an automaticswitched optical network. Fast detection and protection activation solutions developedfor 1:1 span protection have then been integrated with GMPLS for implementing OE-based mesh network shared path protection. The latter study has shown that the use ofthe GMPLS-distributed control plane combined with low-cost all-optical equipment allowsthe cost-effective implementation of fast GMPLS shared path protection. With the introduc-tion of the GMPLS control plane to manage low-cost PXCs, the proposed implementationminimizes the required backup fibers and avoids the duplication of expensive IP router dataports. Experimental results showed that shared path protection implementation achieved arecovery time of less than 5 ms when adopting LOL as a basic failure-detection mechanism.
Acknowledgments
This work was supported in part by the Italian Ministry of Education, Universities, and Re-search (MIUR) under the FIRB “Enabling platforms for high-performance computationalgrids oriented to scalable virtual organizations (GRID.IT)” project and by the Network ofExcellence e-photon/ONE.
References and Links[1] Ethernet IEEE 802.3, http://www.ieee802.org/3/.
[2] Gigabit Ethernet IEEE 802.3z, http://www.ieee802.org/3/z/.[3] 10 GigabitEthernet IEEE 802.3ae, http://www.ieee802.org/3/ae/.[4] S. Shin, B. Ahn, M. Chung, S. Cho, D. Kim, and Y. Park, “Optics layer protection of Gigabit-
Ethernet system by monitoring optical signal quality,” Electron. Lett. 38, 1118–1119.[5] G. Hjalmtysson, P. Sebos, G. Smith, and J. Yates, “Simple IP restoration for IP/GbE/10 GbE
optical networks,” in Optical Fiber Communication Conference, Vol. 37 of OSA Trends in Op-tics and Photonics Series (Optical Society of America, 2000), pp. 275–277.
[6] Rapid Spanning Tree Protocol IEEE 802.1w, http://www.ieee802.org/1/pages/802.1w.html.
[7] Resilient Packet Ring IEEE 802.17, http://www.ieee802.org/17/.[8] P. Pan, G. Swallow, and A. Atlas, “Fast Reroute Extensions to RSVP-TE for LSP Tunnels,”
RFC 4090, May 2005.[9] R. Lingampalli, P. Vengalam, A. Banerjee, J. Drake, J. Lang, and B. Turner, “Performance of
path protection in IP/optical core mesh networks with source-routed GMPLS protocols,” inOptical Fiber Communication Conference, Vol. 70 of OSA Trends in Optics and PhotonicsSeries (Optical Society of America, 2002), pp. 430–431.
[10] J. P. Lang, Y. Rekhter, and D. Papadimitriou, “RSVP-TE Extensions in support of End-to-End GMPLS-based Recovery,” Internet draft, May 2004, draft-ietf-ccamp-gmpls-recovery-e2e-signaling-03.txt
[11] T. Otani, M. Hayashi, H. Tanaka, M. Suzuki, E. Takahashi, H. Nakajima, A. Banerjee, E.McGinnis, K. Sage, and R. Trowel, “Field trial of GMPLS controlled PXCs and IP/MPLSrouters using existing network facilities,” in Proceedings of European Conference on OpticalCommunication (ECOC) (2003), Vol. 2, pp. 240–241.
