Utilities Network Managers’s Guide to Reliable … Network Managers’s Guide to Reliable...

www.sunrisetelecom.comUtilities Network Guided to Reliable Communications

Utilities Network Managers’s Guide to Reliable CommunicationsBy Juan Masmela I May 2011

White Paper

Field testing the converged backbone network

for Teleprotection, SCADA, and Smart Grid

Utilities Network Guided to Reliable Communications2

This white paper outlines eight best practices for applying network testing to resolve converged

backbone network support challenges. Utility providers that adopt these practices can better

meet their service level goals for critical, networked applications – while drastically reducing the

risks associated with downtime and/or sub-standard performance.

Communications networks are a strategic asset and play an important role in the successful operation of an energy power grid. As utility providers strive for greater power grid efficiency by using new applications, the demands on communications networks evolve as do the requirements for network testing.

Backbone networks in utilities are converging with a wide range of technologies and media. These converging technologies include neighborhood area networks, wireless access, wireline access, and core optical networks. The backbone continues to increase in complexity, comprised of TDM, SONET, and IP Ethernet architectures. The media used to deliver data has evolved from copper wire to wireless radios, fiber-optic cables, even the power lines themselves.

Traffic requirements in utility communications have evolved from low-bandwidth applications to a combination of low- and high-bandwidth applications. For example, early utility networks were designed to handle voice and low-speed telecontrol data with point-to-point services. Now, backbone networks in utilities are expected to handle not only low-bandwidth, but also high-bandwidth applications such as video and corporate data. More significant is the modernization of the backbone for Smart Grid initiatives, resulting in an aggressive evolution toward IP and Ethernet, driving new converged backbone networks , that require new test tools, particularly in the field.

SCADA, Teleprotection, Automatic Meter Reading (AMR), Advanced Meter Infrastructure (AMI), Outage Management System (OMS), Work Force Management Systems (WFMS), Automatic Vehicle Location (AVL), and Distribution Automation (DA) are just a few of the key mission critical applications that utilities rely upon for their daily operations. Mission critical applications require high availability, low delay, and fault tolerant networks. More than ever before, these converged backbone networks must be tested thoroughly to work as advertised.

With an increasingly complicated mix of legacy and modern architectures to support, how can utility network engineers be equipped to handle the installation, troubleshooting, and maintenance issues encountered in the field? Clearly, centralized and network-embedded test capabilities are helpful but incomplete. An effective strategy for increasing reliability and performance has to include comprehensive, field-portable testing. This document is a best-practices guide to help utilities increase the reliability and performance of converged backbone networks.

www.sunr isetelecom.com 3

Utility Applications

A utility’s operations and maintenance team can be challenged with supporting a diverse set of applications associated with the converged backbone network. Each of these applications may present variable traffic characteristics, diverse performance requirements and reliability requirements, as well as the obstacles faced when supporting legacy environments – see Table I.

Among the legacy, current, and emerging applications, there are four general classes:•Teleprotection•SCADA•SmartGrid•Enterprise(corporate)

ADR – Automated demand response HS – Hub and spoke (Source: Bell Labs)EMS – Energy management system P2P – Peer to peerP(H)EV – Plug-in (hybrid) electric vehicle SCADA – Supervisory control and data acquisition

TeleprotectionTeleprotection equipment is used by utilities for safeguarding the power distribution network. Also know as Remedial Access Schemes (RAS), Teleprotection schemes are used to resolve faults in the power distribution network, and make an important contribution to preventing voltage breakdown at the customer side. Teleprotection equipment is typically situated at disparate points in the network, and is configured to communicate in order to exchange their measurement values. Communications are transmitted over the backbone network. The equipment is often located near high voltage areas, so to avoid electromagnetic and high frequency induction and ground potential rise (GPR) into the telecommunication network, optical interfaces are used. Delays in Teleprotection commands or degradation in the transmission and security of these protection and remedial action protocols can lead to unexpected and extended power outages, impacting end- customer’s service, creating ill will and negative publicity.

SCADAThe requirements of a SCADA (Supervisory Control and Data Acquisition) data transmission system vary greatly from established transmission systems. SCADA applications are tasked with transmitting mission-critical data, so transmission must be particularly reliable. The connected systems are often distant from one another and housed outdoors (e.g. street cabinets). Also, as is the case with public telecommunications networks, Ethernet and Internet protocols (IP) are becoming more important in SCADA communications links. Consequently, the backbone network must converge to transmit both in order to ensure seamless migration.

Smart GridSmart Grid is a model for transforming the electric power grid by using advanced communications, automated controls, and other forms of information technology. The utility industry is entering a period of significant transformation. Generation, transmission, distribution, and control infrastructure are aging, while energy consumption is increasing. Smart metering and other demand-side techniques have become increasingly necessary to control surges in use during peak and off-peak hours. Industrial scale wind and solar power plants are being connected to the grid as part of worldwide efforts to reduce carbon emissions. Smaller-scale micro-generation solutions, in the form of small wind turbines and photovoltaic cells are being deployed in homes and enterprises. Introduction of alternate and renewable sources of energy and new storage


technologies is fundamentally altering the centralized power generation and distribution paradigm that predominates today. Furthermore, variations in the output power of renewable sources caused by changes in weather and time of day are driving the control of distribution networks to greater granularity. The Smart Grid direction integrates energy infrastructure, processes, devices, information, and markets into a coordinated and collaborative process which allows electricity to be generated, distributed, and consumed more effectively and efficiently. A high performance, reliable and secure communication network is one of the fundamental building blocks to the introduction of Smart Grid applications.

Enterprise (Corporate)Like all other corporate entities, utilities saw benefits in building enterprise networks that would let people throughout the organization exchange e-mail, talk on the phone, and work together using collaborative software. An enterprise network connects all the isolated departmental or workgroup networks into an intra-company network, with the potential for allowing all computer users to access any data or computing resource. This has largely been achieved with IP, Ethernet, and Web technologies that provide better results at lower cost with fewer configuration problems than the enterprise computing models. Originally separate, enterprise data and voice networks are being consolidated onto the converged backbone network, as the interest in Voice over IP (VOIP) and associated IP services increases.

Evolution to Converged Backbone Networks

Whether referred to as Smart Grid modernization, or just as updates to the utilities’ communications infrastructure, the clear trends across the backbone network have been the migration from legacy electrical and copper T1-type infrastructure to optical fiber. The trend from analog to digital is prevalent, and more specifically on the digital side, from TDM or circuit switched to packet switched (IP) and Ethernet.

From Electrical to All-OpticalElectric utilities have been converting electrical connectivity to fiber optic connectivity in most parts of the network for decades. So it’s little wonder that many utilities are also looking to optical networking to help enable their Smart Grid initiatives. Fiber can be the most reliable network transport. Optical fibers do not have ground paths and are immune to noise interference, which eliminates the data errors common to electrical connections. The final piece to an all-optical infrastructure has been the physical connection to Teleprotection equipment, which is most efficiency solved with the IEEE C37.94 interface.

IEEE C37.94 Optical Data Interface

The IEEE C37.94 defines a standard for transmission of n x 64 kbit/s signals via optical interfaces between Teleprotection equipment and multiplexers. In the past, access multiplexers haven’t supported the IEEE C37.94 interface. To connect Teleprotection equipment with MUXes, an additional converter electrical EIA to optical C37.94 has been used. The disadvantage of this solution is that the operation status of the connection between the MUX and the Teleprotection equipment is unknown. This can be resolved by integrating an IEEE C37.94 interface module on the drop side of the Teleprotection access mux, which many vendors now offer (see potential configurations in Figure 1).

Figure 1. Fiber Optic IEEE C37.94 Configurations for Teleprotection Communications


From TDM and SONET to IP and EthernetWith the rapid convergence of data, voice and video, more utilities are migrating from TDM systems and serial communications to Ethernet, for simplicity, scalability, and better interoperability between Control Rooms and different types of substation intelligent electronic devices (IEDs). An important part of this migration is integrating legacy and Teleprotection devices onto an Ethernet-based converged backbone network.

Figure 2. Modern Substation Communications Using TDM over IP/Ethernet

TDM Over IP/Ethernet

TDM over IP has been around for more than a decade. As the technology gets adopted worldwide and as Ethernet/IP continues to be the emerging trend in modern substation communications, TDM over IP will serve as the migration path. As a result, substation backbone network equipment vendors are now developing what is called TDM-over-IP Access Multiplexers. This type of networking gear can integrate TDM DS0 legacy interfaces (such as Voice, RS‐232, RS‐422, RS‐485, V.35, C37.94 and T1 interfaces) with a Layer 2 Managed Ethernet switch that also includes GigE at the WAN side. This equipment allows the transport of critical data such as SCADA, Teleprotection and Relaying with minimum delay, high level of dependability and the zero to minimum down‐time for which TDM-based networks are recognized.


Best Practices for Converged Backbone Network Testing

Following are eight best practices for applying network testing to a utility’s converged backbone network management challenges. Utilities that adopt these best practices will be significantly better able to fulfill their service level goals for all their critical networked applications – while drastically reducing the risks associated with downtime and/or sub-standard performance.

Planning and Assessment

1. Clearly define service level goals. Network service level metrics (SLMs) need to be properly designed from the beginning. Network SLMs are the primary building blocks of an overall quality of service (QoS) strategy. Service levels not only include QoS for the network itself, but also for the network operations and support function. Example QoS metrics for support include: installs without returns, mean time to respond/diagnose/repair, ticket issues post network change.

2. Determine a robust testing strategy. To meet the requirements of high reliability and performance in converged backbone networks, a full complement of test capabilities are necessary throughout the deployment life cycle. Centralized testing systems, embedded test capabilities in network equipment, field-portable test sets and laboratory test fixtures: each have a crucial and complementary role in a robust strategy to find problems before application performance is affected, and to resolve problems quickly when service is, in fact, affected. Field-ready test sets have become more powerful, and through portability, have become the most versatile testing component across planning, installation, and ongoing maintenance stages.

Verification and Installation

3. Validate your vendor’s claims. In a competitive selling situation, equipment or service vendor’s performance claims can sometimes be exaggerated. Performance specifications can be potentially measured under “ideal” conditions. Every utility’s network environment is different. Proactive testing to verify network service and equipment performance really helps eliminate much bigger problems which impact production networks. A thorough verification of key performance metrics of network service and equipment at the point of activation ensures the network is set up right the first time, eliminating returns for troubleshooting.

4. Test the network with assessment tools. Before you start rolling out a converged application like VoIP, you need to know that it can be supported by the network. Start with an end-to-end baseline assessment of the entire network that will handle voice and data traffic, LAN and WAN, including existing bandwidth, latency, jitter and utilization. This includes qualification of physical connectivity, particularly if it’s optical fiber. Review the backbone network’s switch and multiplexer environment, including availability of QoS (Quality of Service) features, the existing and planned network topology and additional network-segmentation requirements. Also, perform load testing to see if the current infrastructure can handle the expected amounts of mixed traffic at the same time. Real-time, end-to-end monitoring and performance testing are the only way to preserve the level of networked application QoS.

5. Create birth certificates at installation. Field test equipment used at network service and equipment activation can create reports with test results and demonstrate that the network element delivered, meets service level goals committed. These tests can also serve as a baseline that can be saved for future reference (i.e., the “birth certificate”). Such birth certificates become a powerful aid in troubleshooting difficult site problems, as baseline deviations can quickly point out problem causes. A straightforward process step can be added to any new installation to the backbone network: create and store a birth certificate as a required final step of the installation.


Table II. Typical Service Level Metrics for Converged Backbone Network Topologies

Monitoring and Troubleshooting6. Plan for the expected – and unexpected.

Doesn’t it feel like a guy named Murphy was a network manager and came up with his law while maintaining and testing networks? Seriously, there are few network managers who wouldn’t agree that planning for the unexpected is essential. Particularly in the field, wasted time adds up quickly if the engineer doesn’t have all the tools required to complete the job. Here is also where an overlapping complement of test capabilities becomes so valuable. For example, there are many types of important off-network / out-of-service tests that a field portable test set can perform and that a centralized test fixture cannot. Also the concept of portable, modular and multiservice testing means a complete tool box is always available for the field engineer to address any issue.

7. Don’t play the blame game. Things will go wrong. But if the separate IT groups (and vendors) involved in a converged backbone network are working together, it’s not so easy to simply point the finger at the network corporate group or network production group or applications group. It is imperative that different groups not only understand what the other is doing, but that they are also working in tandem and regularly communicating. Define specific roles for each group and cross-train the employees. The multiple cultures may remain at odds, but they need to work together effectively. With a converged backbone network, open communications is important. Often, network corporate, network production and applications groups are separate in organizations. But a converged network forces them together.

8. Continue applying above best practices as part of a change management process. To maintain reliability and performance over time, utility IT organizations must incorporate the above best practices into their change management practices. This is essential for ensuring that changes in the converged backbone network - the addition of new sites, new applications, staff relocation – will not adversely affect end-to-end QoS. Such changes can easily affect backbone network performance in totally unexpected ways, so the only safeguard against the interruption of critical application capabilities is to test under the new projected conditions before in production service (and in a test bed). It’s important to note that, while a network test bed will pay for itself by virtue of its support for convergence alone, this technology has many other uses that deliver additional return. These uses include the development of more “network-friendly” applications, better planning of network moves and data center consolidations, and improved support for M&A activity.


Role of Field TestingUtility IT organizations are expanding their use of portable handheld test equipment because of the crucial role that field testing plays in the reliability of and performance of converged backbone networks.

Important maintenance and troubleshooting procedures like comprehensive performance load verification at installation and troubleshooting, can only be done with out-of-service testing in the field. The ability to sectionalize down and isolate individual network components is most effectively done in the field with portable test sets. Test sets give field engineers and technicians immediate visibility into both in-service monitoring and out-of-service diagnostics. When the network is being built, provisioned and operated from an installation and maintenance perspective, it is extremely important that full testing is done at installation to ensure new network elements are working right the first time.

For example, in turning up an Ethernet link over fiber, the utility engineer must make sure the actual throughput delivered is as expected, validated with stats for quality of service initially, and then having the capability to monitor is as necessary. Even when fiber installation is handled via a contractor, utilities find optical test equipment useful. Utilities typically test the fiber on the spools to make sure that there’s no damage from a liability perspective before the contractors go ahead and install the fiber out in the network.

Even given a utility’s test requirements for converged backbone networks, there may still exist a perception that there are simply too many different pieces of test equipment to carry into the field, and that the high combined cost of this equipment, plus the training required to have each field engineer competent with each test set, makes thorough testing ineffective from a cost perspective. Fortunately, cost-wise modern field test equipment designed for a utility’s requirements exists today.

Field Test Set RequirementsWith the migration in utilities to converged backbone networks, support engineers in the field need a multi-service, rugged field-handheld test solution supporting every network interface from legacy to modern IP Ethernet for access, feeder, and core segments. The industry’s most popular solution is the MTT handheld from Sunrise Telecom. First, let’s review the key requirements of such field test sets. The requirements are best sorted into four categories.

Mobile Field ReadinessA field test set for converged backbone networks has to be designed specifically for portability in the field in potentially harsh utility environments:

•Handheldergonomics•Lowsize,weight,longbatterylife•Harshweatherandtemperatureresistant•Hardenedformfactor–provenruggedness•Userinterfacedynamics–easiertousesetupandoperation•Screencontrol–goodoutdoordaylightandlowlightoperation

Multi-Interface AccessA field test set for converged backbone networks has to connect and test at all the potential network interfaces in the utility’s environment:

•OpticalEthernettests–10/100/Gig•OpticalTeleprotectionC37.94tests•SONET–OC1,OC3,OC12,OC48tests•Opticalfiber–OTDRtests•CopperEthernettests–10/100/Gig•TDM–NxDS0,DS1,DS3tests•Electrical232,X.21V.35,422,449,530•Voicetests–2-/4-wire

Multi-Layer TestingA field test set for converged backbone networks has to test at the physical, datalink, and network layers than define the backbone, whether the network uses private and/or public network services. With the advent of IP and Ethernet, reliable high speed networks has been becoming packetized, and actually require tests performed at layers 1, 2, and 3, even though the backbone is effectively acting as a TDM-like transmission network. Likewise, the field test set needs to be able to test public network services completely to may be used with the backbone topology, e.g. carrier Ethernet services.


Modular UpgradabilityA field test set for converged backbone networks has to easily and economically support the network as it evolves. For example, Smart Grid and other modernization initiatives will continue to drive change in utility networks. A modular approach to field test equipment (e.g. with field swappable test interface modules) allows for fast support of technologies and protects existing tool investments.

Field Test Use CasesSunrise Telecom’s MTT utility field test platform is the utility industry’s premier handheld test set for converged backbone network installation, verification, and troubleshooting. The MTT platform includes an upgradeable modular design, which means dramatically lower costs when compared to purchasing separate dedicated test sets. With over 50,000 deployed in the field, the MTT platform has been the industry’s de facto standard for legacy and modern network testing. Modules are available for utility network testing needs and applications, including Ethernet, SONET, TDM, optical C37.94 datacom, and electrical datacom.

The following is a graphical summary guide of the potential network test configurations for the MTT platform to increase the reliability and performance of converged backbone networks.

The MTT utility field test set can isolate and completely verify key network equipment that comprises the converged backbone network. SONET multiplexers and microwave radios can be tested across their line side and drop side interfaces (see left side of Figure 3). The emerging class of TDM-over-IP Access Multiplexers can also be thoroughly tested. (see right side of Figure 3).

Figure 3. MTT Field Test Applications for Converged Backbone Network Equipment

End-to-End Add-Drop Test

SONETC37.94 OpticalDatacomT1/T32/4 W Voice

EthernetEthernet

C37.94 R-232EIA530/V.35T1/T32/4 W Voice

C37.94 RS232EIA530/V.35T1/T32/4 W Voice

(1) SONET MUX/Radio Tests (2) TDM-Over-IP Access MUX Tests

OC1OC3

OC12OC48

OC1OC3

OC12OC48

OpticalLoop Test

ElectricalLoop Test

End-to-End Test

EthernetC37.94RS232EIA530/V.35T1/T3

EthernetC37.94 RS232EIA530/V.35T1/T3

EthernetC37.94 RS232

EIA530/V.35T1/T3

10/100Gig

EthernetLine-SideLoop Test

Drop-SideLoop Test

The MTT can perform additional critical optical network testing. The MTT can connect to either side of optical C37.94 equipment, e.g. standalone converters, and/or MUX modules (see left side of Figure 4). As well, the MTT has an OTDR module to locate fiber breaks, identify faults, estimate fiber attenuation, and verify back reflection. This eliminates the need to carry a separate OTDR test set into the field (see right side of Figure 4).

Figure 4. MTT Field Test Applications for Optical C37.94 Equipment and for Optical Fiber Testing

(3) C37.94 MUX/Converter Test (4) Fiber OTDR Tests

C37.94 Optical

Line-SideLoop Test

Drop-SideLoop Test


C37.94

OTDR Test

Optical FiberOptical Fiber

C37.94

DatacomT1/Nx64k

RS232EIA530/V.35T1/Nx64K2/4 W Voice

RS232EIA530/V.35T1/Nx64K


The MTT platform is a vital tool for core backbone network testing, whether the backbone is based on SONET, Ethernet, or a combination thereof. The MTT handheld test set is the industry’s smallest, combining Gigabit Ethernet testing and SONET testing to OC48 (see left side of Figure 5). The MTT can also be used to perform in-service monitoring of key backbone communication links, monitoring for quality and performance (see right side of Figure 5).

Figure 5. MTT Field Test Applications for Core Backbone Testing and Monitoring

End-to-End Test

(6) Network Monitoring(5) Network Backbone Tests

In-Service Monitoring

SONET, Ethernet,DS3/1

SONET, Ethernet,DS3/1

OC1OC3

OC12OC48

10/100/GigE

OC1OC3

OC12OC48

10/100/GigE

OC3OC12OC4810/100/GigE

DS1DS3OC3

OC12OC48

OC3OC12OC4810/100/GigE

SONET,Ethernet

OpticalLoop Test

OpticalLoop Test

SONET,Ethernet

In addition to testing modern network interfaces, links, and equipment, the MTT platform supports complete legacy network testing including TDM equipment (left side of Figure 6) and analog voice tests (right side of Figure 6).

Figure 6. MTT Field Test Applications for Legacy Network Equipment Testing

DS3/DS1

Line-SideLoop Test

Line-SideLoop Test

Drop-SideLoop Test

Drop-SideLoop Test


DS3DS1

DS3DS1

SONET

DS3DS1

Datacom2/4 Wire VoiceFract T1/DSO

C37.94EIA530/V.352/4 W VoiceFract T1/DSO

RS232EIA530/V.352/4 W VoiceFract T1/DSO

End-to-End Add-Drop TestDS1DS3

SONET

2 Wire Voice4 Wire VoiceVolP Ethernet



(8) Voice Tests(7) T1/TDM Channel Bank Test

PBX


Conclusions

Today, utilities are aggressively pursuing new backbone network options that will result in improved operational efficiency and increased productivity as well as prepare them for Smart Grid deployment. This means that many parameters must be kept in balance. First, networks must be highly reliable: in a mission-critical environment, no compromise is acceptable. Second, capital expenditures (CAPEX) and operating expenditures (OPEX) must be minimized. Finally, the network should offer the opportunity to implement new applications in a rapid and cost-effective manner.

Converged backbone networks can offer at least the same level of reliability, QoS and security as that of traditional utility communications networks, while supporting the full array of both TDM and SONET and new IP/Ethernet services that are needed for core utility operations and administration. New technologies provide utilities with the opportunity to migrate traditional applications to more efficient IP and Ethernet technologies, and to implement new IP-centric applications. It is expected that IP and Ethernet will be key to converging backbone networks for the Smart Grids.

All this impacts the increasingly important role of network testing, particularly in the field. The importance of testing across all deployment phases grows as the need for reliability and performance increase. Field testing complements other network diagnostic capabilities being built into the networks. The key requirements for use of field test equipment in utilities include proven ruggedness, comprehensive network interface support, multi-layer network verification, and clear upgradability to support new network services. With the rapid evolution of Smart Grid, field test set modularity is ever more important. There will be more new interfaces to support, while protecting existing tool investments. Utility network engineers don’t want to carry multiple types of test sets into the field. Modular equipment answers these concerns.

Sunrise Telecom’s MTT utility field test platform is a single, truly handheld solution for the industry’s unique need for modular, multi-service testing of converged backbone networks. Used worldwide, the MTT platform provides the same level of testing performance for utilities and Smart Grid networking as for telecommunication service providers and their networks. The MTT provides the mobile field readiness, multi-interface access, multi-layer testing, and modular upgradability that have been the vital requirements for utility provider’s unique environments.

© 2011 Sunrise Telecom Incorporated. All rights reserved. Specifications subject to change without notice. All product and company names are trademarks of their respective corporations. Do not reproduce, redistribute, or repost without written permission from Sunrise Telecom. C_0210 REV. A01 May 2011

302 Enzo Drive San Jose, CA 95138 • Tel +1 408-363-8000 • Fax +1 408 363-8313

Date post:	22-May-2018
Category:	Documents
Upload:	tranthu
View:	215 times
Download:	2 times

Utilities Network Managers’s Guide to Reliable … Network Managers’s Guide to Reliable...

Documents