Lessons from Field Damage Assessments about ...users.ece.utexas.edu › ~kwasinski › 1569715143...

© Alexis Kwasinski, 2013 1

Lessons from Field Damage Assessments about Communication Networks Power Supply and

Infrastructure Performance during Natural Disasters with a focus on Hurricane Sandy

Alexis Kwasinski

The University of Texas at Austin

February 3, 2013


Extended Abstract

This paper summarizes observations about communication infrastructure performance obtained from field damage assessments made by me after notable recent natural disasters and explores ways of reducing communication networks outages intensity and duration. This discussion is made within the context of observed effects of Hurricane Sandy on communication networks from damage assessments performed shortly after Sandy made landfall. Some of the other recent notable natural disasters which are considered here based on field damage assessments observations made by the author include hurricanes Katrina (2005), Dolly, Gustav and Ike (2008), Irene (2011), and Isaac (2012), and the Feb. 2010 earthquake and tsunami in Chile, the Feb. 2011 earthquake in Christchurch, New Zealand, and the March 2011 earthquake and tsunami in the Tohoku Region of Japan.

Power outages are one of the main causes of communication network outages during natural disasters and Hurricane Sandy was no exception. Reports issued by Verizon indicate that 300 central offices were affected and information collected during my damage assessment indicates that the majority of them were affected by power outages but did not loose service. Unlike Katrina, there is no indication of central office outages caused by engine fuel starvation originated in diesel procurement and delivery issues. Still, two key central offices in Lower Manhattan had at least some of their operations, if not all, interrupted when flooding damaged power backup equipment including onsite diesel generators and fuel pumps in their basements or first floor. These issues could have been prevented by locating power equipment on higher floors or by using watertight doors instead of conventional doors to access the central office buildings. Watertight doors are found in many of Japan’s NTT central offices. In some sites—e.g. Kamaishi—these doors were effective in limiting the damage during the March 2011 tsunami. Power was also the main reason why an average of 25 % of the cell sites in the area affected by Sandy lost service. During the damage assessment I observed that the majority of base stations in cities and towns in the area affected by Sandy are located on buildings roofs and have no permanent gensets. This is different from what is observed in the Gulf Coast where most cell sites are placed on their own parcel and many of them have permanent gensets. Placing base stations on buildings roofs led to power restoration issues because although some of these cell sites have standard power sockets at the ground level and with easy access to connect a portable diesel generator, many sites did not have any pre-prepared way of connecting a generator at ground level so a cable had to be run from the roof to a portable genset on the street. This ad-hoc solution led to service restoration delays because of difficulties in gaining building or roof access. Significant issues were found due to communication networks users increased need for reliable power, e.g. to charge their cell phones.

Recently, more comprehensive solutions to address power issues have been proposed. In recent years, implementation of smart grid technologies has been seen as a way of improving electric grids performance during natural disasters. Although some of these technologies allow a faster identification of electric outages and general grid condition, power grids damage repairs still requires human intervention. Moreover, the damage assessments and statistical outage data indicate that power grids are inherently very fragile systems due to their centralized power distribution and control architectures. Since most utility-based smart grid technologies still maintain such centralized architectural and control approaches, these smart grid enabled power supply availability improvements are limited. Network operators-based solutions and, in particular, microgrids—independently controlled and confined power grids with their


own local power generation sources, such as Verizon’s Garden City central office that operated during Sandy powered by 7 fuel cells—may present a better technical solution. Moreover, for high-power loads, such as central offices, microgrids are cost competitive with respect to conventional power solutions. Still, operation of some microgrid local power generation units may be affected during disasters because of their dependency on other infrastructures, such as roads for diesel supply. Some of the solutions to address dependency on other infrastructures include using technologies that have low probability of being affected by a given hazard—e.g. natural gas in hurricane prone areas—diversifying local power supply technologies, using local energy storage devices—e.g. batteries—and using renewable energy sources that do not depend on other infrastructures, such as photovoltaic (PV) systems or wind generators. Yet, relatively large footprint and output variability limits the application of PV and wind generators. Limited space and higher relative cost makes implementation of alternative power solutions even more difficult in distributed network elements: base stations, outside plant fiber nodes, such as digital loop carrier (DLC) terminals, and CATV amplifiers supporting telephony services. Although fuel cells have been deployed in some cell sites and natural gas generators have been installed in many DLC terminals along the Gulf Coast, different city architecture approaches makes such solutions difficult to reproduce in the Northeastern Atlantic Coast, leading to unsafe ad-hoc solutions observed with Sandy, such as placing portable generators on top of pole-mounted CATV amplifiers.

Although the discussion focuses on power supply issues, problems related with other communication networks infrastructure components are also commented. For example, the discussion also explores the implications in terms of network vulnerabilities of using fiber optics remote terminal cabinets to restore service to copper cable facilities that were damaged due to a combination of pressurization failures and flooding in the aftermath of several natural disasters (including Sandy).


DISCUSSION I. INTRODUCTION

This paper discusses general findings about communication systems infrastructure performance observed during field damage assessments performed by the author. These observations are placed into context with preliminary findings made during damage assessments performed in the area affected by superstorm Sandy. These damage assessments were conducted from November 2 to November 5 and from November 22 to November 23, 2012 in the area affected by Hurricane Sandy. Tracks for these trips are shown in Fig. 1.

Hurricane Sandy made landfall as an extra-tropical cyclone (cold core) with maximum sustained winds of about 80 to 90 mph, equivalent to that of low-intensity Category 1 hurricane according to the Saffir Simpson Scale. However, Sandy’s unprecedented large size generated a storm surge found in more intense hurricanes (when characterized based on the Saffir Simpson Scale that only considers maximum sustained wind speeds) with observed heights of 14 ft in The Battery, NY and Kings Point, NY. These high storm surge levels flooded and damaged some areas in Lower Manhattan, New Jersey’s barrier islands, and narrow coastal strips of New Jersey’s Atlantic Coast, Staten Island, Long Island, and Long Island Sound and shorelines along the Hudson and Hackensack rivers.

The rest of this paper is organized as follows: First, the impact of superstorm Sandy on communications systems is discussed and findings about infrastructure performance are compared with observations made in previous notable disasters. This fist part of the paper is divided into observations made for different types of communication networks: traditional wire-line telephony networks, wireless communications networks, cable TV (CATV) and other digital telephony networks, data centers, and other communication systems. Since the first part of this paper identifies power supply issues as a main cause of communication services disruptions in all observed natural disasters, including Sandy, the second part of this paper discusses general aspects of power grids performance during natural disasters. This second part also discusses power options for critical loads, particularly communication sites. Finally this paper concludes by summarizing the main points made in this paper and presenting photographic evidence that supports the observations made in the discussion.

II. WIRE-LINE TELEPHONY NETWORKS Although communication networks did not experienced outages as extensive as those observed with

Hurricane Katrina [1], issues similar to those observed with Hurricane Ike still were still found. However, due to the higher population and network elements density in area affected by Hurricane Sandy and its large storm size, the effects of this storm on communication systems were more noticeable than those observed with Hurricane Ike. Reports issued by Verizon indicate that 300 central offices (COs) were affected and information collected during the damage assessment indicates that the way these COs were affected was through loss of electric grid power supply. Still, although the majority of them were affected by power outages they did not loose service. This is a common outcome in most disasters, even the most intense ones. For example, during the March 11, 2011 earthquake and tsunami in Japan, approximately 1,000 of NTT’s 1,800 buildings in the disaster region were affected in different ways but mostly due to power outages [2]. Initial issues affecting most of these approximately 1,000 buildings were soon addressed and less than 1% of COs lost service. Still, on March 28th 55 COs were still presenting issues.


Sixteen of these 55 buildings were having minor damage, but the site was not under normal operation due to damage to the power plant but not to the switch fabric, or due to some other unknown reason. Of the remaining 39 COs, 26 were destroyed by the tsunami or all of their equipment had been rendered useless by the tsunami, 4 were isolated due to damaged fiber optic links, and 9 could not be reached because they were located within the forced evacuation area around the Fukushima #1 Nuclear Power Plant. Another example can be found with Hurricane Katrina. As Fig. 2, shows, although there were some destroyed COs, the majority of the line outages were related with power issues that originated in two causes: CO unreachable due to flooding and genset fuel starvation (which, ultimately, is a logistical-related problem). However, unlike with Katrina, with Sandy there is no indication of outages at central office caused by engine fuel starvation originated in diesel procurement and delivery issues.

Still, two key central offices in Lower Manhattan flooded and at least some of their operations, if not all, were interrupted when flooding damaged power backup equipment including onsite diesel generators and fuel pumps in their basements or first floor. One of these facilities in Lower Manhattan is Verizon’s main building located at 140 West Street. As Figs 3 and 4 show, water was still being pumped out of this site on November 3. Figure 4 also show a large containerized mobile diesel generator used to power part of this building. Flooding also affected the main cable entrance facility and likely damaged cable pressurization equipment used to prevent multi-pair copper cables to corrode due to moisture or water entering the cable. Evidence of the likely damage to the cable pressurization equipment is found in Fig. 5 showing emergency cable pressurization equipment units. Even when water may not have damaged cable pressurization equipment, air pumps for cables would not have worked due to lack of power at the site. With the main entrance facility flooded and without pressurized air, water entered the cables. Although there are procedures to dry the cables of water and moisture, these methods are not perfect. Hence, higher failure rates can be expected in these cables. This same problem was experienced in New Orleans after Katrina [1]. At that time, Bellsouth used the opportunity to modernize their copper cable plant and installed more fiber optic cables connected to digital loop carrier (DLC) remote terminals (RTs) [3], such as the one in Fig. 6. These fiber optics-based systems were also used to restore service in the areas of COs destroyed by Hurricane Katrina [1], such as the one in Figs. 7 to 9 placed where the destroyed COs of Yscloskey, Delacroix and St. Bernard used to be located. This same solution was the one used by NTT to restore service in some COs that had the switch fabric destroyed by tsunami flood waters (Fig. 10). One advantage that these fiber-optic RT present for network planners is flexibility to adapt to an uncertain demand in the long aftermath of a natural disaster. As it is explained in [4] a disaster is characterized by four phases: pre-disaster, during the disaster, immediate aftermath and long-term aftermath. Each of these phases has their specific problems. One of these problems found in the long-term aftermath is that after a natural disaster some areas experience a demand drop due to loss of residences and business caused by natural disaster. This demand may eventually recover or may never reach pre-disaster levels, at it occurred in several areas in New Orleans and the Mississippi River delta after Katrina, where even today the demand is lower than that before Katrina [3]. The planning question in these areas after the disaster struck is whether to restore communication infrastructure to the same capacity level than before the disaster. If copper multi-pair cables are used, then they need to be sized based on the expected demand 5 or 10 years into the future. Since in disaster areas with severe damage in residences and business this expected demand is so difficult to anticipate, restoring service with copper multi-pair cables is a very risky approach. Instead, since at a distribution level a single fiber optic cable can meet all traffic demand


needs, restoring service with fiber-optic cables is from a demand planning perspective less risky than using copper.

However, there are two options to power fiber-optic RTs. One is to use existing or newly installed copper cable facilities in order to power the RTs from a CO with a ± 190 V system [5]. However, this powering approach requires the RTs to be within a few miles of the hosting COs and in many cases it is not possible to use existing copper cable facilities. Hence, in the most common cases when fiber-optic RTs cannot be powered from the CO, the need to be powered locally. Hence, in the most common approach, fiber-optic RTs are powered locally from the grid. In order to power RTs during grid outages, RTs are typically engineered to accommodate enough batteries to provide 8 hours of backup [5]. However, environmental conditions—specially heat—limit batteries autonomy and life. Still, during long power grid outages caused, for example, by a natural disaster, a few hours of battery backup is not sufficient to avoid RTs service loss. That is, the chance of loss of service during natural disasters for subscribers served from RTs is higher than that for subscribers served with a traditional wireline telephony copper-based architecture. This observation is verified in several past recent hurricanes [5]. For example, in the aftermath of Hurricane Ike an important number of outages for AT&T were originated in loss of service in 551 DLC RTs. Of these, less than 3 % of the failures were caused by equipment destruction, such as the RTs in Fig. 11. The rest of the failures occurred when the batteries discharged during the long power outages. Statistics from Hurricane Hugo in 1989 shows similar values than those with Ike in 2008 with 555 DLC RTs loosing power but only 10 destroyed. In 2003, approximately 800 DLC RTs lost power after Hurricane Isabel. Many DLC RTs had power issues in 1992 after Hurricane Andrew, too when more than 1000 DLCs lost power, 722 of them in Miami. Usually, the solution for this problem is to deploy portable gensets, such as those in Fig. 12, however, during hurricanes or intense storms unsafe conditions do not allow deploying the gensets before batteries are discharged. Nevertheless, ingenious approaches are found in the Gulf Coast with infrastructure improvements implemented after Katrina [3]. Some of these vulnerability reduction approaches seen for DLC RTs include placing them on elevated platforms that are equipped with a permanent natural gas generator (e.g. Fig. 6). This solution proved to be very effective during hurricanes Gustav and Isaac, although in some areas, lack of natural gas distribution networks, obligated to follow the conventional approach of deploying a portable genset (Fig. 12). Still, use of elevated platforms and natural gas generators may not be possible in Manhattan due to practical reasons related with the significant architecture differences between both areas and although in Manhattan it is possible to power RTs from the CO with a ± 190 V power distribution system, the cables of this power distribution system would have been damaged in the same way that copper multi-pair telephony cables were damaged in lower Manhattan. In general, the performance of already existing DLC RTs during Sandy was similar to those in aforementioned hurricanes. With Sandy, reports from Verizon indicate that up to 600 DLC RTs or other digital wireline communications equipment both in vaults and in cabinets (Fig. 13) were affected by Hurricane Sandy; some due to flooding and most due to power outages.

Figures 14 and 15 show the other central office that flooded in Lower Manhattan, this one located at 104 Broad Street. These two photos show mobile diesel generators on both sides of the building and two diesel trucks on the Water Street side. The damage assessment was able to identify two other central offices that may also likely flooded and had damage to power back up equipment: one in the Rockaway


Peninsula (Fig. 16) and in Long Beach (Fig. 17 Left). All these four central offices likely experienced some loss of service.

Although Verizon attempted to prevent these sites from flooding by using sandbags, this solution seems to not have been very effective. Flood damage and loss of service could have been prevented in a number of other ways. One solution is to place onsite diesel generators and other power plant components, including fuel pumps, on higher floors. This was the design that likely prevented the central office at Lavallette (north of Seaside Heights) in Fig. 17 (Right) to have its diesel generator damaged even when Sandy’s storm surge flooded the streets around the building. In some occasions, building structural resistance is mentioned as a concern when placing power plant components on higher floors. However, good construction practices prevent such potential structural issues. One notable example is shown in Fig. 18. These photos show the CO building in the town of Onagawa, Japan. Even when this CO was one of NTT’s buildings located closer to the March 11, 2011 Mw 9.0 earthquake epicenter and was also completely submerged by the tsunami, Fig. 18 shows that the there were no structural damage where the batteries were located, in the power room located on the first floor. Another good approach observed during the damage assessments in the aftermath of the March 11, 2011 earthquake and tsunami in Japan, is shown in Figs. 19 and 20 for the COs of Kamaishi and Unosumai, Japan. The figure corresponding to the CO in Kamaishi exemplifies a construction practice also observed in other COs: use of watertight doors. In addition, these and other CO are also surrounded by a wall with flood gates in order to provide limited flood protection for CO buildings. Although in some locations in Japan this design did not prevented damage from the tsunami because of its tremendous magnitude, watertight doors and flood gates reduced the damage in Kamaishi and other locations [6] and it would have prevented flood waters to enter the aforementioned buildings affected by Sandy. All other visited central offices in the affected area did not show flood damage, mostly because significant flood waters did not reach other sites and sandbags protected the building from the limited potential flood waters that reached those other building (Fig. 21). The only other central office that requires a special mention is the one in Fig. 22 in Garden City. This central office has seven 200 kW fuel cells that can power the facility when fueled by natural gas. When this site was visited on November 3, there was still no commercial power on, yet this site showed no signs of being out of service suggesting that the fuel cells or other onsite generation mean where operating adequately.

Wireline telephony outside plant equipment was also affected by flooding and power outages or direct damage. Issues with DLC RTs were already discussed. In some specific few limited areas fires also caused damage to the outside plant infrastructure (Fig. 23). Flooding also affected manholes (Fig 24), particularly in lower Manhattan. Although poles were damaged, loss of poles was limited even in the areas affected by the storm surge where the damage to homes was most severe. In some of these areas it was usually possible to observe that the pole closest to the shore had been damaged (Figs. 25 and 26) either by houses floating away (Fig. 25) or by other debris caused by the intense waves. Still, in many cases with significant damage to homes, damage to poles and telephony lines was light or non-existent (Fig. 27).

III. WIRELESS COMMUNICATION NETWORKS

Wireless communication networks also presented issues with Sandy. Reports indicated that an average of about 25 % of the base stations in the affected area lost service—some network operators


informed of a lower percentage of base stations out of service. During the damage assessment on November 3 to 5, wireless communications coverage in the western half of the Rockaway Peninsula was almost non-existent (Fig. 28) Cellular coverage also showed issues along the New Jersey coastline. The cause of loss of service in almost all of these affected base stations was lack of power. During the damage assessment it was possible to observe that the majority of base stations in cities and towns in the affected area were placed on buildings roofs and had no permanent gensets. Some of these cell sites (Fig. 29) have standard power sockets located at ground level and with easy access so a portable diesel generator can be transported to the site and be easily connected. However, many sites (e.g. Fig. 30) did not have any way of connecting a generator at ground level. In these sites a cable had to be run from the roof to the street where the portable genset was located. In practice, placing these generators in service may take considerable time because of roof or building access practical issues, such as in some cases the need to locate the building owner or administrator in order to coordinate building access. This may have been the reason why in some sites, gensets were not deployed until a few days after the storm (Fig. 31). Like in all previous hurricanes (Fig. 32), with Sandy genset deployment was not coordinated among network operators leading to the inefficient logistical approach of having multiple gensets in each of several cell sites (Fig. 33) because each network operator deployed and refueled a generator for its own based station at a shared cell site instead of coordinating a single genset deployment among all network operators sharing a cell site. With increased number of gensets per cell site, it is expected that traffic in roads and diesel demand are increased, and that more gensets may fail. Still, portable gensets was the most widely used technological approach chose to power cell sites during the long power outages that followed Sandy. This was also the approach observed after Katrina and Ike. However, like it happens with DLC RTs, hurricane-caused flooding (Fig. 34) and high winds may prevent network operators to deploy portable gensets before batteries are discharged and the base station looses power. In order to address these limitations, in the past few years, permanent propane-fueled gensets have been widely deployed in the Gulf Coast. Use of these propane generators has proved to be effective in the aftermath of hurricanes like Ike, Gustav, and Isaac, provided that the propane tank is not placed at ground level where it can be damaged by the storm surge, like Figs. 35 (left) exemplifies. Another alternative observed in some few cells in the Gulf Coast area is to use fuel cells, such as the one in Fig. 35 (right), although it is unclear why in this site the base station was also powered by a portable genset instead the fuel cell. Some possible explanations are difficulties in distributing hydrogen cylinders in a disaster area or cost advantages in powering a load with the diesel genset instead of using the fuel cell.

Although with Sandy most cell sites power issues, some, like the one in Fig. 36 may have been damaged by storm surge. This photo exemplifies a construction practice difference between the northeast Atlantic Coast and the Gulf Coast: in the former region base stations that are not located on building rooftops are not placed elevated above the flood plane as it is done in many cells sites of the Gulf Coast. Still, it is possible to observe that in the Gulf Coast this practice is not followed uniformly because it is also possible to find cell sites like those in Fig. 37 where base stations had been placed at different heights.

In order to provide coverage in some areas with base stations out of service, network operators deployed cell on trucks (COLTs), such as those in Figs. 38 to 40, and cell on wheels (COWs) such as that in Fig. 41. Several of the COLTs have satellite links (Figs. 38 to 40). Although most of these COLTs and COWs were deployed shortly after the hurricane passed, at least one was deployed on Nov. 3 (Fig. 40).


Use of COLTs and COWs is comparable to what was observed in damage assessments after past hurricanes with the exception of use of satellite links which seem to have been used more often in the aftermath of Sandy than during the restoration process in previous hurricanes, such as Katrina or Ike.

Neither the damage assessment nor other sources of information have identified issues in mobile telephony switching offices (MTSOs). Location planning for wire-line telephony COs represents a vulnerability disadvantage of wire-line telephony networks with respect to wireless communication networks because MTSOs location is not necessarily constrained by distance limits between wire-line telephony subscribers and their hosting CO. Since the connection between MTSOs and their hosted cell sites can be established with fiber-optic or microwave radio links, the MTSO can be located in sites with reduced risk of flooding. In the case of a wire-line telephony CO, such choice is not always possible. Use of fiber-optic RTs in wire-line telephony may reduce these planning constraints for wire-line telephony CO sitting and, as it was explained, it would shift vulnerability concerns onto powering issues on fiber-optic RTs which makes the vulnerability condition analogous to that in wireless communication networks where the concern is placed on powering cell sites. However, shifting vulnerabilities into distributed network elements (wire-line fiber-optic RTs and wireless communication cell sites) has the advantage of reducing the number of users affected by a communication outage because a loss of service in one distributed element involves fewer users than those hosted by a CO or MTSO. Moreover, in some situations, loss of service in a given base station could be overcome by increased coverage of neighboring cell sites that are still operational. This strategy was widely used to restore service in areas served by cell sites destroyed by the tsunami of March 11, 2011 in Japan, by increasing network coverage of base stations located on surrounding hills and that were not reached by the tsunami, such as the cell site in Fig. 42. Still, since in wireless communications since cell sites are used as repeaters for fiber-optic or microwave radio links for cell sites that are farther away to the MTSO, in some situations the loss of a cell site may affect more users than just those in that particular affected cell site. One solution to this issue is to provide geographical diverse paths for the fiber-optic or microwave radio links. However, in some cases, such solution may be excessive costly. Since the most significant vulnerability in most cell sites is in its power supply, another solution implemented in some wireless communications networks is to create a hierarchical categorization for the cell sites based on their importance with respect to transmission links to the MTSO and provide longer battery reserve time (up to 12 or 24 hours) to those cells with higher priority and shorter battery reserve time (4 hours) to those cells at the end of the transmission path. In the specific case of Sandy, the damage assessment provided no indication of severed links to cell sites, as it was previously observed with Katrina, Ike and the earthquake and tsunami in Japan. In all these cases service was restored through mobile microwave radio repeater units (Fig. 43). Evidently, the vulnerability advantage of wireless communication networks over wire-line telephony networks in terms of switching centers sitting is lost if wireless communications switch fabric or centralized call/data management components are collocated with wire-line switching equipment. In the case of Sandy, there is no indication that collocation may have led to wireless communications networks issues as it happened during Katrina (Fig. 44) and the earthquake and tsunami in Japan (Fig. 45)

One important issue in the operation of communication networks after Sandy that was also associated to the extensive power outages caused by the storm was the need for charging stations for cell phones and other personal communication devices. That is, Hurricane Sandy’s aftermath highlighted an increasing vulnerability of communication networks as their architectures are more distributed and more relying on


power from an electric grid, particularly at the end user level. In order to address this need both Verizon and AT&T provided charging stations where people could charge their phones (Figs. 46 and 47). However, it is unclear how many people were unable to communicate due to difficulties in charging the batteries. Private organizations also provided charging stations, in some cases powered by renewable energy, such as the one in Fig. 46.

IV. CABLE TELEVISION (CATV) AND OTHER DIGITAL TELEPHONY NETWORKS Powering distributed network elements and users communication devices is also an important issue

for cable TV (CATV) operators, who often also offer phone services routed through the Internet. In order, to power their distributed network elements, CATV operators used in some cases small camping off the shelf gensets placed on their equipment cabinets on top of poles (Fig. 48). Besides the obvious safety issues (high winds affected the disaster during a northeastern about a week after Hurricane Sandy made landfall), it is likely that these poles had not been calculated to support the extra load. The photovoltaic (PV) modules on poles shown in Fig. 48 suggest a possible alternative to power CATV network elements in disaster conditions. However, these PV modules need to have a dedicated connection into the CATV equipment or, if the PV module is integrated into the (public) power grid, the inverter that is equipped with the PV module does not need to be of the grid-tie type, because in the United States and according to IEEE Standard 1547 this type of PV inverters are required to shut-off when the power grid to which they are connected is experiencing a power outage. One alternative to power these pole mounted CATV equipment that was observed in the Gulf Coast is to use pad mounted natural gas gensets, such as the one in Fig. 49 observed after Isaac. However, ground location is at risk of damage from storm surge as the site in Fig. 50 exemplifies. In this site, the pad mounted generator was destroyed by Katrina. As a result, after Isaac this CATV node was powered by placing a gasoline portable generator on top of the pole mounted cabinet. Use of camping-type portable gensets placed on poles was widely observed not only after Isaac (Fig. 51) but also after Gustav (Fig. 51).

V. DATA CENTERS Internet use has also become a critical need both for everyday life and during disasters. In many

cases, Internet is also currently used to transmit telephony services, too. Several data centers experienced issues and loss of service during Hurricane Sandy. A few of them, such as those in 75 Broad Street, 33 Whitehall Street, 325 Hudson Street, and 121 Varick Street reported flooding and damage to generators and fuel pumps that led to loss of service. Partial loss of service due to a generator malfunction was reported at the data center in 111 8th Avenue (Fig. 52). Several private data center lost service due to engine fuel starvation caused by difficulties in diesel procuring and delivering. Despite limited flooding, no loss of service was reported in the data centers at 25 Broadway and 32 Avenue of the Americas. Loss of service was also avoided in the important data center located at 60 Hudson Street (Figs. 53 and 54). One of its diesel generators was still operating when this site was visited on November 3. It is unknown why this site had two relatively small portable diesel generators outside and with cables running into the interior of the building.


VI. OTHER COMMUNICATION SYSTEMS Broadcasting networks (radio and TV) also experienced service affecting issues during Hurricane

Sandy. One example is shown in Fig. 55 where the site with the transmission towers was flooded and the huts with the transmission equipment were damaged.

There are no information of damage to a few intercontinental cable landings on New Jersey’s coast.

VII. POWER GRIDS PERFORMANCE AND POWERING OPTIONS As shown, power supply is one of the main causes of communication network outages during natural

disasters. The strong correlation between power and communication outages is also verified through statistical analysis, for example, presented in [7] for the 2010 earthquake in Chile and represented in Fig. 56 for the 2011 earthquake in Japan. As it is exemplified in Fig. 57 with Hurricane Ike, power grids are extremely fragile systems in which damage to fewer than 1 % of its components—e.g. 1 pole in a line of 100 poles (e.g. Fig. 58)—causes extensive outages that could affect all or nearly all electricity users in a county. Even, in areas where there is predominately extensive damage, it is also possible to see inhomogeneous damage intensity with little damage a few hundred meters away from intense damage. As it is also exemplified in Fig. 57, during hurricanes intense power grid damage is limited to coastal areas affected by storm surges. Still, even in normal operation, in the U.S. power grids availability—the expected portion of time when a power grid is operating with respect to the total evaluated time that equals the sum of the time when the power grid is operating and the time when it is not operating—is about 0.999 (often indicated as 3-nines availability). That is, without considering the effects of natural disasters, it can be expected that power grids in the U.S. operate during 99.9 % of the time, which is the same to say that the total outage time for power grids may likely add up to last a little less than 9 hours in a year. The expected average power outage restoration time is about 2 hours. It is important to emphasize that power grids availability of 3-nines is calculated without considering outages caused by natural disasters or any other outage that last significantly longer than most common outages. Still, even without considering these longer than usual outages occurring during so-called “major event days”, 3-nines availability is insufficient for communication systems which require individual outage times shorter milliseconds and an overall system availability of 5-nines—i.e., communication systems are expected to operate at least 99.999 % of the time.

As it is explained in [8] and [9], in order to achieve the desired availability of 5-nines from the 3-nines provided by the power grids, communication power plants are equipped with batteries and gensets, in a typical power plant architecture shown in Fig. 59. When a site is equipped with gensets that are in standby until a power grid outage occurs, the availability observed at the output of the rectifiers is of 4-nines. The remaining availability increase up to 5-nines is provided by batteries. That is, the only reason why communication network operators equip their sites with a backup power plant is because power grids are unable to provide a sufficiently high availability level required by communication loads. Moreover, although U.S. power grids have availability levels superior to most countries in the world, the availability of power grids is low enough to make the addition of gensets alone insufficient and make several hours of battery energy storage essential to reach the required power supply availability level. Still, as Fig. 59 shows, air conditioning systems cannot typically be powered from the batteries, which means that the cooling circuit of telecommunication sites have an availability level of at best equal to the availability of

the combination of the power grid and an onsite standby genset or 4-nines [10]. In order to achieve 5-


nines availability, network operators need to rely that the heat load and the site thermal inertia is sufficient to limit the temperature rise long enough to allow time to get the genset repaired or the power grid restored before the equipment shuts down due to high temperatures. However, these availability levels, such as 3-nines for the power grid with a power outage restoration time of 2 hours, do not apply during disaster conditions. Thus, during disasters, power supply chain and thermal circuit in communication sites is expected to be below the 5-nines availability minimum considered under normal conditions.

In order to address the problem of low power supply availability during natural disasters there are two approaches: focus on the real source of the problem which is the power grid fragility or accept that power grids fragility could not be sufficiently improved and search for network operator solutions. A commonly suggested approach to reduce power grids fragility is to bury cables and some substations. However, besides the much higher cost of underground infrastructure, although underground power infrastructure is less likely to fail during hurricanes, when components fail repairs will take longer than in power grids with overhead infrastructure. Notice that since power grids in a given area typically operate most of the time in normal conditions when faults are also possible, the result of having underground facilities is longer outage restoration times even during normal conditions. Moreover, underground power facilities are not exempt of damage as it was observed with Sandy when flooding affected electric utilities manholes or during the earthquake of February 2011 in Christchurch, New Zealand when soil liquefaction damaged many buried power cables. Improvements can also be made in overhead power grids by trimming tree branches or reinforcing poles. However, these improvements increase power grids availability during disasters in a limited way because the inherently vulnerabilities of power grids—centralized control, and centralized power generation and distribution architecture—are not changed with these improvements. Recently, more comprehensive solutions to address power issues have been proposed. In recent years, implementation of smart grid technologies has been seen as a way of improving electric grids performance during natural disasters [8]. Although some of these technologies allow a faster identification of electric outages and general grid condition, power grids damage repairs still requires human intervention. Moreover, these technologies still rely on a centralized power distribution and control architectures that make power grids to be fragile systems. Since most utility-based smart grid technologies still maintain such centralized architectural and control approaches, these smart grid enabled power supply availability improvements are also limited.

The alternative to address power grids fragility through power grid-based approaches is to implement solutions at the network operators’ level. The conventional approach implemented by network operators is to still rely on backup power plants and increase diesel storage tank capacity and/or battery backup time. But, as it was discussed, these solutions are also limited. For example, adding batteries does not improve the availability of the cooling circuit so outages due to high temperatures equipment shut down are equally likely, as it is verified by cases of CO outages during the 2010 earthquakes in Haiti and Chile where at least one CO in each event failed due to cooling issues. The more modern approach that could be implemented by network operators is to power communication sites with microgrids, such as those introduced in [11] and [12]. As it is represented in Fig. 60, microgrids are independently controlled and confined power grids with their own local power generation sources. These local power sources are the main power source of the microgrid and when they are combined with a local controller, allows the load—i.e. the communications equipment—to be powered independently of the power grid condition. Although the power grid can be used as a power source for the microgrid, it is a secondary power source.


Although these local sources make microgrids to be a relatively costly solution, for high-power loads, such as COs, microgrids are cost competitive with respect to conventional power solutions. Moreover, use of absorption chillers in combined-heat and power cycles addresses the aforementioned cooling circuit low availability problem [10]. As a result, with an adequate design and planning, microgrids could reach availability levels of 6 nines or higher [13] [14]. To date, two microgrids operated by communication network provides, one in Sendai and the other in Garden City (Fig. 22) have operated satisfactorily during natural disasters.

The key planning aspect of microgrids focuses on the choice for local power generation unit technologies [13] [15]. Local power sources can be classified in two types: those that rely on a lifeline—another infrastructure, such as natural gas distribution networks, essential for their operation—and those that do not require a lifeline. The most common technologies of local power generation units requiring lifelines include reciprocating engines, microturbines and fuel cells (with local reformers). Evidently, during disaster conditions, the issue with these generation technologies is that their lifelines may also be affected by the natural disasters as the power grid is. Since the way in which lifelines are affected in a natural disaster depends on the disaster type and intensity—e.g. natural gas networks are not significantly affected by hurricanes but they are likely to suffer significant operation disruptions during earthquakes—a risk assessment is essential during the microgrid planning process in order to choose the right local power unit technologies [4]. One way of reducing the impact of lifeline dependency is to use diverse power source technologies relying on different lifelines. Still, the main approach to reduce the effect of lifeline dependency is through local storage. For example, if the local generators are microturbines fueled by natural gas, dependency on natural gas distribution networks is reduced by storing natural gas at the microgrid site or by connecting batteries to the microgrid distribution grid. That is, lifeline dependency on an energy source can be reduced through local energy storage [16]. The reserve time of these local energy storage assets allow measuring the degree of dependency on the lifeline [16]. The other type of local sources that do not depend on lifelines are, essentially, renewable energy sources; mostly photovoltaic (PV) and wind generators. Although renewable energy sources are good choices to power communication sites in microgrids during disasters because PV and wind generators do not rely on lifelines, they have two important limitations: relatively large footprint—i.e. low power density in comparison with communications equipment power density—and semi-stochastic power output. The solution to these two issues is analogous to the solutions implemented for sources depending on a lifeline: add local energy storage and diversify the type of renewable energy sources used—i.e. combine PV and wind generators. For the interested readers, mathematical analysis and insights about microgrids planning can be obtained in [13].

Another approach involving the use of microgrids aims at changing the paradigm of managing power and communications traffic separately presented in [17] proposes creating so-called sustainable wireless areas (SWA) formed by a few base stations—e.g. 7 base stations—that are interconnected in a microgrid powered by renewable energy sources. In the SWAs issues with renewable energy sources are reduced by allowing a more distributed layout of renewable energy sources that are not necessarily confined to the cell sites parcel but, more importantly, management of local power generation assets and energy storage is integrated with cell sites traffic shaping. That is, traffic and impact of service quality at the base stations within the SWA are shaped so that the power consumed by the base station can be adapted based on the projected renewable power availability and energy storage levels. That is, cell sites and power


management become essentially integrated. In a related approach presented in [18] and applicable to data centers, modular distributed data centers (MDDCs) are primarily powered by local power sources, particularly but not limited to PV and wind generators. That is, each of these MDDCs forms a microgrid. These MDDCs are expected to be geographically dispersed in a large area or among different regions of a country or the world. Hence, (if) when power is not enough at a given site, another site in the “cloud” with enough power and idle data capacity takes over the tasks of the site with insufficient generated power. Interconnection among these sites is made primarily through fiber optics cables, which have a cost at worst of a few tens of thousand dollars, i.e., 100 times less than a transmission line and is not affected by natural disasters as much as power lines. That is, photons transmitted through fiber optic cables are used as a “proxy” for electrons that, otherwise, would be circulating though transmission lines. In this way, data and power portions of the information and communications technology systems become integrated into a single operational entity. These sites may have local energy storage to provide more operational flexibility both for power generation and for data storage used as part of site hand-over processes.

VIII. CONCLUSIONS

This paper has discussed the effects of superstorm Sandy on communication networks and compared their performance with that observed during previous recent notable natural disasters. The discussion is based on extensive photographic evidence collected during damage assessments shortly after these disasters occurred. The study points out that power supply issues are a major source of communication network disruption during natural disasters. Thus, this paper also discusses power supply alternatives including smart grid technologies and microgrids. References: [1] A. Kwasinski, W. W. Weaver, P. L. Chapman, and P. T. Krein, “Telecommunications Power Plant Damage Assessment for Hurricane Katrina - Site Survey and Follow-Up Results,” IEEE Systems Journal, vol. 3, no. 3, pp. 277-287, September 2009.

[2] A. Kwasinski, “Effects of Notable Natural Disasters from 2005 to 2011 on Telecommunications Infrastructure: Lessons from On-Site Damage Assessments,” INTELEC 2011, 9 pages, Amsterdam, Netherlands, October, 2011.

[3] A. Kwasinski, “U.S. Gulf Coast Telecommunications Power Infrastructure Evolution since Hurricane Katrina,” International Telecommunication Energy Special Conference, 2009, 6 pages, Vienna, Austria, May 10-13, 2009.

[4] A. Kwasinski, “Technology Planning for Electric Power Supply in Critical Events Considering a Bulk Grid, Backup Power Plants, and Micro-Grids,” IEEE Systems Journal, vol. 4, no. 2, pp. 167-178, June

2010.

[5] A. Kwasinski, “Telecommunications Outside Plant Power Infrastructure: Past Performance and Technological Alternatives for Improved Resilience to Hurricanes,” in Proc. INTELEC 2009, pp.1-6.

[6] A. Kwasinski and A. Tang, “Telecommunications Performance in the M=9.0 Off-shore East Coast of Japan Earthquake and Tsunami, March 11, 2011,” Japan Society of Civil Engineers’ International Symposium on Engineering Lessons Learned from the Giant Earthquake, Tokyo, Japan, March 4, 2012.

[7] L. Dueñas-Osorio and A. Kwasinski, “Quantification of Lifeline System Interdependencies after the 27 February 2010 Mw 8.8 Offshore Maule, Chile Earthquake,” in press Earthquake Spectra.


[8] A. Kwasinski, “Implication of Smart-Grids Development for Communication Systems in Normal Operation and During Disasters,” INTELEC 2010, 8 pages, Orlando, FL, USA, June 6-10, 2010,

[9] A. Kwasinski, “Analysis of Vulnerabilities of Telecommunication Systems to Natural Disasters,” 2010 IEEE Systems Conference, pp. 359-364, San Diego, CA, USA, April 5-8, 2010.

[10] A. Kwasinski, “Analysis of Electric Power Architectures to Improve Availability and Efficiency of Air Conditioning Systems,” 2008 International Telecommunications Energy Conference (INTELEC), vol. 10, no. 2, pp. 1-8.

[11] A. Kwasinski and P. T. Krein, “A Microgrid-Based Telecom Power System using Modular Multiple-Input DC-DC Converters, in Proc. INTELC 2005, pp. 515-520.

[12] A. Kwasinski and P.T. Krein, “Optimal Configuration Analysis of a Microgrid-Based Telecom Power System,” in Proc. INTELEC 2006, pp. 602-609.

[13] A. Kwasinski, V. Krishnamurthy, J. Song, and R. Sharma, “Availability Evaluation of Micro-Grids for Resistant Power Supply During Natural Disasters,” IEEE Transactions on Smart Grid, vol. 3, no. 4, pp. 2007- 2018, Dec. 2012.

[14] A. Kwasinski, “Quantitative Evaluation of DC Microgrids Availability: Effects of System Architecture and Converter Topology Design Choices,” IEEE Transactions on Power Electronics, vol. 26, no. 3, pp. 835-851, March 2011.

[15] A. Kwasinski and P. T. Krein, “Telecom Power Planning for Natural and Man-Made Disasters,” 2007 International Telecommunications Energy Conference (INTELEC), pp. 216-222.

[16] A. Kwasinski, “Local Energy Storage as a Decoupling Mechanism for Interdependent Infrastructures,” 2011 IEEE International Systems Conference, pp. 435-441.

[17] A. Kwasinski and A. Kwasinski, “Architecture for Green Mobile Network Powered from Renewable Energy in Microgrid Configuration,” to be presented at IEEE Wireless Communications and Networking Conference (WCNC) 2013, April 2013, Shanghai, China.

[18] A. Kwasinski, “Technological assessment of power supply options for communication sites operating in grid-islanded environments,” in Proc. IEEE INTELEC 2012, 8 pages.

Photographic evidence and other supporting figures.

Fig. 1. Damage assessments tracks. Background map: © Google


Fig. 2. Bellsouth failed CO areas due to Hurricane Katrina indicating the outage cause.

Fig. 3. Water being pump out through the main entrance of Verizon’s central office at 140 West Street.


Fig. 4. Right: Water being pumped out and a diesel generator running at the entrance on Barclay Street for Verizon’s main central office in Manhattan. Left: Flooding on Barclay St. from water being pump out of

Verizon’s central office at 140 West Street.

Fig. 5. Emergency cable pressurization units at 140 West Street. The image on the right shows the

pressurized air injection nozzle.

Fig. 6. A DLC RT in New Orleans installed a few months after Hurricane Katrina. Notice the natural gas

generator.


Fig. 7. Left: Remains of Yscloskey CO after Hurricane Katrina. Right: Same site after Hurricane Isaac

Fig. 8. Left: Remains of Delacroix CO after Hurricane Katrina. Right: Same site after Hurricane Isaac.

Notice the yellow portable genset deployed to power the RT on the platform.

Fig. 9. In the background the building of Bellsouth’s former St. Bernard CO flooded by Katrina.

Equipment inside this building was never replaced and parts of its functions were restored with the DLC RT shown in the foreground and installed after Katrina.


Fig. 10. Two DLC RTs uses to partially restore service in the area of NTT’s Onagawa CO that was

submerged by the tsunami.

Fig. 11. DLC RT destroyed by Hurricane Ike. The one on the right had been installed after Hurricane Rita

in order to partially restore service lost when Sabine Pass’ CO was destroyed.

Fig. 12. Portable gensets deployed to power DLC RT after Isaac. The photo on the right shows both a pre-

Katrina RT at ground level and a post-Katrina RT on a platform with a natural gas generator. It is unknown why a portable diesel genset was deployed at this site.


Fig. 13. A vault (left) and a DLC RT cabinet (right) in the Rockaway Peninsula.

Fig. 14 Pearl Street side of Verizon’s central office at 104 Broad Street.

Fig. 15. Water Street side of Verizon’s central office at 104 Broad Street.


Fig. 16. Verizon’s central office at Rockaway Beach. The photo on the right shows a large portable genset

behind the waste liquids vacuum pump.

Fig. 17. Left: Verizon’s central office in Long Beach. Right: Verizon’s central office in Lavallatte. Notice

the genset located on the second floor.

Fig. 18. Onagawa CO Building. Left: Exterior. Right: Interior; power room.


Fig. 19. Watertight door in Kamaishi, Japan.

Fig. 20. Flood gate in Unosumai central office, Japan.

Fig. 21. Verizon’s central offices in Brigantine (Left) and Wildwood (Right). Notice in both cases the

sandbags at the doors.


Fig. 22. Fuel cells at Verizon’s central office in Garden City.

Fig. 23. Damage from a fire affecting communications infrastructure.

Fig. 24. Left: A Verizon’s truck working next to manhole. Right: A manhole still with water.


Fig. 25. A severed telecom cable hanging from the pole on the right.

Fig. 26. Damage to the pole nearest the sea in Ortley Beach

Fig. 27. Important damage to dwellings in Union Beach but little damage to poles.


Fig. 28. A person talking on a satellite phone standing on top of a truck with solar panels to provide

electricity to people in the Rockaway Peninsula.

Fig. 29 A rooftop cell site in Ausbury Park powered by a genset at street level. Notice the standard

connector for the genset.


Fig. 30. A rooftop cell site in the Rockaway Peninsula.

Fig. 31. Another cell site in the Rockaway Peninsula

Fig. 32. Multiple portable gensets deployed to cell sites in the aftermath of hurricanes. Left: Katrina.

Right: Ike.


Fig. 33 A cell site with multiple genset (3 portable and one permanent) at the intersection of I-95 and I-

195 in New Jersey after superstorm Sandy.

Fig. 34. Flooding around a cell site near after Hurricane Isaac.

Fig. 35. Power backup alternatives for cell sites in the aftermath of Hurricane Isaac. Left: Propane tank

damage by Isaac’s storm surge. Right: Fuel cells.


Fig. 36. A cell site in a marina near Seaside Heights. The boats were scattered around by Sandy’s storm

surge.

Fig. 37. Two examples of cell sites in New Orleans with base stations located at different heights.

Fig. 38. A COLT in front of the Freedom Tower in New York City (left) and a Colt in the Rockaway

Peninsula (right).


Fig. 39. Left: A COLT near Seasigh Heights, NJ. Right: Barclay Street in NYC with a COLT parked next

to Verizon’s central office at 140 West Street.

Fig. 40. Two COLTs in Long Beach. The one in the back was being deployed when this photo was taken

on 11/3/2012.


Fig. 41 A COW along the Garden State Parkway.

Fig. 42. A cell site in Rikuzentakata, Japan, destroyed by the March 11, 2011 tsunami.

Fig. 43. Microwave repeater units linking cell sites. Left: in Biloxi, MS after Katrina. Right: in Otsuchi,

Japan, after the March 2011 tsunami.


Fig. 44. Map showing outage causes in wireless communications networks after Katrina.

Fig. 45. NTT Docomo (wireless) switching equipment at NTT East’s (wire-line) Nobiru CO.

Fig. 46. COLTs and a charging station in Long Beach.


Fig. 47. An emergency communications truck in Rockaway Peninsula

Fig. 48. CATV equipment in Bayonne, NJ.

Fig. 49. A pad mounted natural gas generator powering CATV equipment after Hurricane Isaac.


Fig. 50. Left: A pole mounted CATV amplifier and power supplier with a pad mounted natural generator destroyed by Katrina’s storm surge. Right: The same CATV equipment after Isaac powered by a camping

portable genset placed on top of the CATV amplifier and power supply

Fig. 51. Examples of camping-type portable gensets used to power CATV pole mounted equipment. Left:

After Isaac. Right: After Gustav.

Fig. 52. Data center building at 111 8th Avenue.


Fig. 53. Data center at 60 Hudson Street. Notice the diesel generator exhaust on the photo on the right.

Fig. 54. Two views of the 60 Hudson Street data center showing additional small gensets in the area and a

truck with diesel to replenish the fuel tanks at this site.

Fig. 55. Transmission towers for WEPN, damaged by Sandy.


Fig. 56. Evolution in time of power grid outages as a per unit of maximum number of customer outages and communications outages during the 2011 earthquake and tsunami in Japan. Notice that due to the presence of energy storage, communication outages peak about a day after power outages reach their

maximum.

Fig. 57. Left: Maximum percentage of electricity users without power with respect to total number of electricity users in a given county of Texas. The shaded counties represent counties with less reliable

outage statistical data that those in non-shaded counties. Right: Percentage of power grid infrastructure components damage observed during damage assessments conducted after Hurricane Ike.

Fig. 58. Left: Power grid outage condition in Grand Isle on 9/2/2012. Red traces indicate outage lines. The dot indicates the point where the photo on the right was taken. Map source: Entergy. Right: Photo

taken on 9/2/2012 in the location indicated by the dot on the map on the left. The photo is looking northeast and shows that the power outage in the island is caused by damage in only one pole of many.


Fig. 59. Representation of a conventional power plant for a communications site.

Fig. 60. Representation of a microgrid showing its main components.

Date post:	04-Jul-2020
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

Lessons from Field Damage Assessments about ...users.ece.utexas.edu › ~kwasinski › 1569715143...

Documents