Countdown to ShutdownIgnoring the Aging Wireline Switching Infrastructure at Our Own Peril

2

AbstractThe world’s roughly 40,000 legacy telephone switches still support the same number of lines that they did before the onslaught of mobile phones – about 900 million worldwide. Yet the phone services provided by these switches support a substantial portion of earnings for many of the world’s largest communications companies. These switches continue to be vital operating components in many countries’ critical infrastructure, enabling key communications services for con-sumers, businesses, first responders, health care, and military and governmental entities.

However, this critical infrastructure is on a rapid “countdown to shutdown.” Engineered for replacement prior to their 20-25 year end-of-design-life, most switches have passed that mark and are now operating on borrowed time. Because switches have performed in the past with reasonable regular-ity, operators can be easily lulled into believing that they will continue operating today exactly as they did yesterday. But just like relying on an aging football player to sustain the team, this past performance should not be an indicator of future reliability. Worse yet, the “countdown to shutdown” is exacerbated by the attrition of skilled employees to operate the switches, vendors who are no longer able to support them, and a diminishing supply of spare parts.

In order to understand the substantial risk of continued operation, operators need to visualize the critical trajectories impacting these switches including age, assumptions about longevity, line loss, and planned replacement rate. This simple effort should involve network planners, financial man-agers, and Chief Risk Officers. Operators may find that, at the time their switches effectively expire, they will leave many thousands of customers stranded and incur a loss of revenues that poses serious detrimental consequences to their profit-ability and sustainability.

Once they visualize their critical trajectories, operators can develop a path to mitigate this risk, including implementing a Switch Status Evaluation, Internal Resource Assessment, Transformation Plan, and Outage Contingencies. Operators should partner with vendors who have the support resources and appropriate products and solutions to cost-effectively and rapidly transform this aging, high risk switching infrastructure to IP technology.

It’s easy to forget about infrastructure. As long as it’s there, we go about our daily lives without thinking much, if at all, about it.

But when important infrastructure fails, it can have enormous and life-changing repercussions. The broken levees in New Orleans during Hurricane Katrina and the collapsed I-35W bridge in Minnesota are testament to that fact. The problem of aging infrastructure, however, goes beyond things like roads, bridges and levees. It also applies to out-dated telephone switches.

Legacy telephone switches, commonly known as “TDM Switches”, have far outlived their designed product lifecycles, yet these elderly pillars of our communications infrastruc-ture continue to bear the load of wireline telephone traffic. Nobody keeps track of the statistics any longer, but the last counts of TDM switches indicate that about 40,000 are still in operation globally, serving about 900 million lines.1 Using the U.S. as a more discrete example, about 13,000 switches are in operation today serving around 100 million lines – roughly the same level of lines as in the mid-1990s, before the onset of widespread use of mobile devices. But don’t assume these are all “lines for grandma.” Indeed, as residential line loss continues, the percentage of active lines remaining on these switches is increasingly weighted toward high-value business customers, and many are not lines at all but are instead high usage business trunks like PRIs.

We’re all committed to a packet-based networking future and, in fact, wireline networks today often employ a mix of newer, IP-based technology and legacy, TDM-based gear. But despite the gradual move to IP, it doesn’t appear that TDM switches are disappearing any time soon, though some dis-cussions are beginning at various national levels. The U.S., for example, has somewhat leapfrogged most other countries by initiating regulatory inquiries to create a final disposition plan for its Public Switched Telephone Network or PSTN. However, implementing nationwide activities to carry out any future plan will need to take into account many different user constituencies, and will therefore likely be a multi-year, maybe even multi-decade, process.

1 Telegeography, 2011

Countdown to Shutdown: Ignoring the Aging Wireless Switching Infrastructure at Our Own Peril

3

So, for the time being anyway, the legacy switches that power wireline networks across the globe – and many bil-lions in service provider revenue – are here to stay. Yet even the telephone experts that own and operate legacy switches can often fail to understand the risks of continued reliance on the TDM switch – leading to an inevitable and possibly highly disruptive “countdown to shutdown.”

That’s a big problem. But it doesn’t have to be.

The inherent risk in continuing to operate these old monoliths is much more tangible, visible, and understandable when op-erators take a few moments to do some fairly simple “back-of-napkin” math to visualize their critical trajectories. By crunch-ing the numbers on TDM switch retirement, operators can lay a path to a sensible, pragmatic transformation of their aging network based on schedules that take into consideration their unique risks and fragile condition.

The alternative, of course, is to simply ignore the problem of legacy switch degradation and to address this issue only

after the force of a disaster, however big or small, compels them to act. In that case, the “countdown to shutdown” may be exceedingly brief, with long term consequences.

The Legacy Switch LifecycleTDM switches have served us well for the past 20 to 30-plus years, so why the urgency around them now?

Most legacy telephone switch infrastructure was put in place when communications was a very different animal, and TDM switches have generally survived far longer than they should have. The life expectancy of these switches was 20 to 25 years, with an expectation common to all types of fixed assets that they would be replaced before they reached their end of design life. Most switches have been in continued, virtually non-stop operation for 25 to 30 years, and there are often no upgrade plans in sight. Indeed, it is almost impos-sible to find any fixed asset in any industry where non-stop operation (200,000 to 300,000 continuous hours) has been expected or relied upon for mission-critical services – without ever undergoing significant refurbishment.

Countdown to Shutdown: Ignoring the Aging Wireless Switching Infrastructure at Our Own Peril

Left to Right: 1. Components resoldered, evidence of repair recycling. 2. Components catching fire. 3. Back plane pins fusing. 4. Ancient chip sets

4

That’s not good news in itself, but it becomes even more problematic when you consider that it will be increasingly dif-ficult to keep these old switches working going forward.

Nearly all switches or switch subsystems have been manufac-turer-discontinued or are at end of life. In the meantime, the age and wear on some of these switches is causing a multitude of daily issues such as fusing of switch backplane pins onto circuit boards, faceplate breakage, and the require-ment for multiple recycling of repairs on critical controller boards.

Component failure should not come as a surprise. The below sample page from the DMS 100 switch operations manual shows that at year 15, the typical chip component on key cards should have died at least once –

and there are thousands of these components in a DMS switch. Legacy switch owners can use third parties to replace faulty board components, of course, but finding reliable components or replacements cards for these switches is tough and getting tougher. And the more scarce these spare parts become, the more expensive they get.

Many operators believe that they can rely on the grey market to provide critical cards for continued operation, but most cards have been recycled through repair and return pro-cesses multiple times. Even if they do survive another power cycling (booting-up of the replacement card on the switch), the switch becomes dependent upon control cards from the late 1970s or early 1980s with hundreds of thousands of operating hours on them. This is comparable to buying rusty old rebar to repair an aging bridge: it’s an improvement of sorts, but it will only slightly delay the bridge’s demise. In addition to the nuts and bolts of the legacy switch prob-lem, you have the human element.

Skilled personnel to operate switches, many of whom were trained in the 1980s, have left or are now leaving the

telephone business in droves as they get older and as the operators phase-out these positions in favor of less expensive individuals with expertise in IP technology. That means there are fewer and fewer technicians around to maintain these highly complex legacy switches. Worse yet, it also means that few will be available to assist the operator in enabling a transition of these switches, lines, and customers to IP technologies.

The clock is also ticking on the ability to turn to outside sup-

Countdown to Shutdown: Ignoring the Aging Wireless Switching Infrastructure at Our Own Peril

Switch component predicted failures in DMS switch manual. Averages one failure per component by year 15.

5

port for legacy switch issues, given that original switch ven-dors are increasingly unable to support switches for extended years. Just like the knowledge base rapidly exiting the opera-tors, vendors are also losing human resources that know how to repair and maintain these ancient, complex systems.

The Value of WirelineAs our legacy wireline switch infrastructure continues to crumble, many operators have turned their attention to that bright shiny object known as the mobile device, and the recent boom in smartphones, tablets, wirelessly-enabled laptops and other connected devices has most of them shifting much of their investment, focus, and CapEx to mobile networks.

Mobile is great, but like all new technology introductions, prod-uct adoption typically becomes a combination of both substitu-tion and complementary usage. A survey authored by the National Center for Health Statistics illustrates that although the move to wireless continues, it is unlikely everything will move to mobile, at least in the foreseeable future. In some segments of the population, such as unrelated adults living in rented accom-modations, wireless substitution is already the norm. In other segments, such as families living in a home they own, wireless substitution is much lower.

The real key to wireline’s continuing value, however, is in the business setting where technology adoption tends to occur rapidly, but technology replacement takes a much longer time. This dynamic is evidenced by operator line losses, where busi-ness line loss rates are often significantly lower than residential.

Business customers are of high value to operators, paying more per line and buying other services like data and Internet. However, when switches begin having significant outages, these high-value businesses are also likely to be the most seri-ously affected. Business customers are therefore more likely to defect following significant outages due to the critical nature of telephony for business, the number of competing alternative providers, and the relative lack of acceptable mobile substi-tutes.

Wireline switches also serve mission-critical functions, for example, in military bases, hospitals, and first responder envi-ronments. When a switch serving a downtown area cannot be repaired for hours, days, or weeks, these customers and those

who are dependent upon their services are likely to experience substantial adverse effects.

That said, not only are incumbent operators’ reputations on the line, so are their bottom lines. The wireline voice network remains a significant contributor of revenue to incumbents providing, for example, about $25 billion of AT&T’s revenue 2.

Despite these very respectable numbers, voice networks are having difficulty staying margin-positive. As noted in a recent TMCnet article, Bernstein analyst Craig Moffett argues that, over the last decade, the returns on invested capital in com-munications networks in U.S. markets have been anemic, at best. “Wireline networks have the weakest returns on invested capital with a 1.5 percent gain over the last decade.” It is also worthy to note that there are few variable costs due to wireline voice being fixed asset-based, so these networks are being hard hit by line loss since little variable cost is associ-ated with them. When a line is lost, much of the revenue loss falls directly to the bottom line.

And it’s only going to get worse. Although lines are decreas-ing, they decline fairly equally across all switches. This means that a switch that carried 18,000 active lines in 2001 may now support only 9,000 active lines. Revenues may be halved, but operating expenses may actually have increased. This gets even worse a decade from now when switches are around 40 years old – assuming they even make it that long – and lines are significantly lower than today’s levels and costs to maintain them are significantly higher. Time is of the EssenceWhat, me worry? This catchphrase, popularized by MAD Magazine’s Alfred E. Neuman, would seem to indicate that there’s no need to worry about things. The delivery may be hu-morous, but clearly, this type of thinking that can lead to peril.

Consider the possibility of what could happen in the event of a legacy switch failure. Just one outage of this sort could result in a large downtown being without wireline voice service for many hours or days, e.g., government buildings totally without service; banks having to contend with mobile alternatives; and legal firms without billable client connections. Such an event could also place severe strain elsewhere in the aging network.

2 AT&T 2011 10K

Countdown to Shutdown: Ignoring the Aging Wireless Switching Infrastructure at Our Own Peril

6

Given the advanced age of switches and impending potential for catastrophic failure, some carriers are looking to regula-tory relief as a means to ease their burden of modernizing this infrastructure. In the U.S., for example, the Federal Com-munications Commission and state public utilities commis-sions are likely to provide some level of regulatory relaxation, but betting exclusively on regulatory relief as a panacea is in itself a high-risk strategy.

Telephony is such a complex service with so many levels of customer need, there is clearly no panacea solution or “get out of jail free” regulatory card.

Globally, legacy telephone lines have been decreasing at an annual 4 percent rate over the last decade reaching about 900 million today, while at the same time mobiles have increased to about 6 billion (http://www.itu.int/ITU-D/ict/facts/2011/material/ICTFactsFigures2011.pdf). Assum-ing the 4 percent decline continues, about 600 million lines will remain by 2022. At the same time, these TDM switches will be entering their fourth decade of constant operation. Irrespective of any optimistic hope for regulatory relief, time is clearly of the essence in establishing a transformation plan for these aging monoliths.

Visualizing is Key to Grasping the Size of the ProblemPerhaps human beings by their nature are procrastinators, sporting an “If it ain’t broke, don’t fix it” approach – un-less they are prompted by some compelling event. Indeed, as you watch the odometer on your 15-year-old car flip to 200,000 miles, that event may become compelling enough to prompt you to replace it. But these ancient switches have no odometers, unfortunately.

Similar to the visual indication of age that an odometer pro-vides, operators should find a way to visualize the relevant trajectories of their fixed line voice businesses. They need to model factors like line loss, switch age, expected switch life, and switch transformation plans in order to adequately assess risks.

Simple, back-of-napkin mathematical risk modeling may help considerably, because as each day passes, operators are

coming closer and closer to hitting a critical mass of switch outage disasters unless they act. As switches begin dying increasingly over this decade, they may expire in such quanti-ties for an operator that it will be highly unlikely they can be replaced or upgraded without substantial external interven-tion and diversion of company resources.

Visualizing it this way: Assume an operator has 100 30-year old switches (with a design life of 25 years) and it be-lieves they will continue operating another 10 years. Today those switches serve a million lines, and assume a 10 percent line loss rate per year and a replacement plan of 5 switches per year. The simple math shows that a decade from now, the 50 remaining switches will die – stranding 178,000 lines and the loss of many hundreds of millions in revenue.

Exercising a Risk Management Course of ActionDoing the simple math is key to visualizing the potential for network risk, and avoiding an uncontrolled, volatile, and potentially damaging countdown to shutdown.

Many operators are simply not updating their switch infra-structure at a clip that will allow an orderly transition of their customers to IP. In any case, for the most part, this risk may still be a few years off, which might make it seem less urgent. But when this risk becomes reality, it is likely to have wide-

Countdown to Shutdown: Ignoring the Aging Wireless Switching Infrastructure at Our Own Peril

Switch Retirement Calculator from http://www.genband.com/calculators/tdm

7

spread effects for our industries, governments, first respond-ers, and residential and business customers.

Operators can immediately take multiple steps to mitigate the risks associated with TDM switch outages:

1. Chief Risk Officer Assessment Chief Risk Officers at the operators should take notice, assess the associated Enterprise Risk (the Enterprise be-ing the operator in this case) using simple mathematical projections, and work with network and financial person-nel to develop plans and actions to mitigate these risks.

2. Trajectory Visualization Operators should develop models to visualize and plan for an orderly transformation of their switching infra-structure. This task requires them to develop projections regarding switch longevity, line losses, migration rates to IP access alternatives, regulatory relief and switch trans-formation plans. This plan should ensure that switches are fully retired and replaced before they expire.

3. Switch Status Evaluation A constant evaluation process should be performed regarding status – switch by switch. This should include all relevant factors: customer and earnings risks, current switch age, switch type, status of modules and com-ponents, location, line counts, customer types, vendor support levels, and availability of spare parts.

4. Internal Resource Assessment One of the most critical factors in a risk mitigation plan is the ongoing availability of skilled employee opera-tional resources. An assessment should be performed to determine what resources are needed to enable the continued effective operation of remaining switches. In addition, transforming each switch involves employees across a wide range of skill sets – e.g., translations, operations, methods, systems – so assessments should assume the need for dedicated resources that can work with vendors to enable the orderly transformation of the switches.

5. Transformation Plan

The transformation plan is the culmination of efforts be-tween the operator and vendor that lays out the orderly process. This includes inputs from all operator levels – marketing, finance, network, CTO, and IT. Operators must work with skilled vendors that have both the sup-port teams and advanced IP solutions to rapidly trans-form these switches and associated infrastructure to IP.

6. Outage Contingencies Every operator should have in place contingency plans that prepare for unexpected switch outages. These plans should include assumptions for outages of portions of the switch for a few hours, to full switch outages over a period of days or weeks.

GENBAND is the global leader in fixed network transforma-tion, having transitioned over 1,200 TDM switches globally to IP technology. With our innovative network transforma-tion portfolio of call control, session border control, unified communications and gateway solutions, GENBAND enables central office cost savings and Average Revenue Per User (ARPU) growth from new IP-based services. In addition, our extensive Professional Service organization has state-of-the-art, field-proven transformation tools and processes as well as millions of hours of relevant field experience.

www.genband.com 1-866-GENBAND© 2012 GENBAND Inc. All rights reserved.The GENBAND logo is a registered trademark of GENBAND Inc. This document and any products or function-ality it describes are subject to change without notice. Please contact GENBAND for additional information and updates.

Countdown to Shutdown: Ignoring the Aging Wireless Switching Infrastructure at Our Own Peril

Switch Retirement Calculator from http://www.genband.com/calculators/tdm