Communication Network Architecture and Design...

◆ Communication Network Architecture and Design Principles for Smart GridsKenneth C. Budka, Jayant G. Deshpande, Tewfik L. Doumi, Mark Madden, and Tim Mew

An integrated high performance, highly reliable, scalable, and securecommunications network is critical for the successful deployment andoperation of next-generation electricity generation, transmission, anddistribution systems—known as “smart grids.” Much of the work done todate to define a smart grid communications architecture has focused onhigh-level service requirements with little attention to implementationchallenges. This paper investigates in detail a smart grid communicationnetwork architecture that supports today’s grid applications (such assupervisory control and data acquisition [SCADA], mobile workforcecommunication, and other voice and data communication) and newapplications necessitated by the introduction of smart metering and home area networking, support of demand response applications, andincorporation of renewable energy sources in the grid. We present designprinciples for satisfying the diverse quality of service (QoS) and reliabilityrequirements of smart grids. © 2010 Alcatel-Lucent.

cells is being deployed in homes and enterprises.

Introduction of alternate and renewable sources of

energy and new storage technologies is fundamen-

tally altering the centralized power generation and

distribution paradigm that predominates today.

Furthermore, variations in the output power of renewa-

ble sources caused by changes in weather and time

of day are driving the control of distribution networks

to finer and finer time scales.

“Smart grid is a concept for transforming . . . [the]

electric power grid by using advanced communica-

tions, automated controls, and other forms of infor-

mation technology. This concept, or vision, integrates

energy infrastructure, processes, devices, information,

and markets into a coordinated and collaborative

IntroductionThe global electric power industry is entering a

period of significant transformation. Generation,

transmission, distribution, and control infrastructure

are aging while energy consumption is increasing.

Figure 1, which was developed using data from the

U.S. Department of Energy [18], illustrates the trend

in worldwide electricity consumption between 1980

and 2006.

Smart metering and other demand-side tech-

niques have become increasingly necessary to control

demand during peak and off-peak hours. Industrial-

scale wind and solar power plants are being connected

to the grid as part of worldwide efforts to reduce car-

bon emissions. Smaller-scale micro-generation in the

form of small wind turbines and photovoltaic (PV)

Bell Labs Technical Journal 15(2), 205–228 (2010) © 2010 Alcatel-Lucent. Published by Wiley Periodicals, Inc. Published online in Wiley Online Library (wileyonlinelibrary.com) • DOI: 10.1002/bltj.20450

206 Bell Labs Technical Journal DOI: 10.1002/bltj

process which allows electricity to be generated, dis-

tributed, and consumed more effectively and effi-

ciently” [13]. A high performance, reliable, and secure

communication network is one of the fundamental

building blocks to the introduction of smart grid appli-

cations.

This paper addresses network architecture and

design principles for an integrated smart grid commu-

nication network. We examine some of the challenges

faced in supporting a diverse set of applications each

with varying network performance requirements, relia-

bility requirements, and traffic characteristics, as well

Panel 1. Abbreviations, Acronyms, and Terms

AC—Alternating currentADR—Automated demand responseAMI—Advanced metering infrastructureBPL—Broadband over power lineCCTV—Closed circuit televisionCDMA—Code division multiple accessCPP—Critical peak pricingCS—Class selectorDER—Distributed energy resourceDiffServ—Differential servicesDSCP—Differential services code pointDSL—Digital subscriber lineEDGE—Enhanced data rates for GSM EvolutionEF—Expedited forwardingEMS—Energy management systemEPRI—Electric Power Research InstituteGPON—Gigabit passive optical networkGPS—Global positioning systemGSM—Global System for Mobile

CommunicationsGtCO2e—Giga (metric) tonne carbon dioxide

equivalentHAN—Home area networkHSPA—High-speed packet accessIEC—International Electrotechnical CommissionIED—Intelligent electronic deviceIEEE—Institute of Electrical and Electronics

EngineersIETF—Internet Engineering Task ForceIntServ—Integrated servicesIP—Internet ProtocolISM—Industrial, scientific, and medicineISO—Independent system operatorL1, L2, L3—Layer 1, 2, 3 (of OSI model)LAN—Local area networkLMR—Land mobile radioLTE—Long Term EvolutionMDMS—Meter data management systemMPLS—Multiprotocol Label SwitchingNAN—Neighborhood area networkNASPI—North American SyncroPhasor Initiative

NASPInet—NASPI networkNERC—North America Electric Reliability

CorporationNIST—National Institute of Standards and

TechnologyOFDM—Orthogonal frequency division

multiplexingOSI—Open System InterconnectionP2P—Peer-to-peerPEV—Plug-in electric vehiclePHEV—Plug-in hybrid electric vehiclePLC—Power line carrierPMU—Phasor measurement unitPRIME—PoweRline Intelligent Metering

EvolutionPTT—Push-to-talkPV—Photovoltaic (cells)QoS—Quality of serviceRF—Radio frequencyRFC—Request for commentsRTO—Regional transmission organizationRTP—Real time pricingRTU—Remote terminal unitSCADA—Supervisory control and data

acquisitionSDH—Synchronous digital hierarchySONET—Synchronous optical networkTDM—Time division multiplexingTOU—Time of use (pricing)UMTS—Universal Mobile Telecommunications

SystemUPS—Uninterruptible power supplyVAR—Volt-ampere reactiveVoIP—Voice over IPVPN—Virtual private networkVVWC—Volt, VAR, Watt controlWAMS—Wide area measurement systemWAN—Wide area networkWiMAX—Worldwide Interoperability for

Microwave Access

DOI: 10.1002/bltj Bell Labs Technical Journal 207

as the challenges faced with supporting legacy appli-

cations and networks. While there are many legacy,

new, and evolving applications, the following five

classes of applications (not necessarily mutually exclu-

sive) will be used as examples in presentation of

communication network architecture and design prin-

ciples in this paper:

• Smart metering, also known as advanced meter-

ing infrastructure (AMI),

• Automated demand response (ADR),

• Teleprotection,

• Distribution automation, and

• Micro grid management.

We begin with an overview of the evolution of a

traditional power grid to the smart grid. Next, we

present a high-level characterization of smart grid

applications including brief descriptions of the appli-

cation examples listed above. A smart grid communi-

cation network architecture is presented including the

physical connectivity architecture, examples of logical

connections, access network options, and the archi-

tectural implications of shared ownership of networks.

We then address specific quality of service (QoS) and

reliability design considerations for integrated smart

grid communication networks. We illustrate the

“green benefits” of a smart grid—and by implication

those of the integrated communication network—and

offer our conclusions and recommendations on areas

for future work.

Complete treatment of smart grids requires dis-

cussion of a wide variety of technologies and topics.

Due to space restrictions, we have had to limit scope.

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

Year1980

Electricity demand is increasing in Asian countries, and in China in particular, which saw demand for energy grow nearly tenfoldover a 25 year span. The North American market experienced a twofold increase over the same period despite drastic reductionin the manufacturing industry and slow population growth. Along with the pent up demand, energy sources are becoming scarceand the cost of generating electricity is becoming prohibitive. Therefore, making efficient use of electric energy should, in theory,help reduce dependence on fossil fuel and combat carbon emissions.

1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 P2006

Ener

gy

con

sum

pti

on

in b

illio

n K

WH

Asia & OceaniaAfricaMiddle EastEurasiaEuropeCentral & South AmericaNorth America

Figure 1.Worldwide electricity consumption.


An essential topic not addressed in this paper is net-

work security—a topic worthy of several papers on

its own. Furthermore, details of local area networks

(LAN) or home area networks (HAN) are outside of

the scope of this paper.

An Overview of Smart GridThere is a wealth of information on the smart grid

concept and its evolution in the public domain. The

most comprehensive smart grid overviews are found

in the 2009 reports to U.S. National Institute of

Standards and Technology (NIST) prepared by the

Electric Power Research Institute (EPRI) [5] as well as

the final NIST Framework and Roadmap document

[17]. These extensive reports draw on contributions

from consensus-gathering workshops attended by

representatives from government agencies, regula-

tors, vendors, (communication) service providers,

academia, and standards organizations. Some of the

earlier EPRI work on IntelliGrid* can be found in [6].

The following brief smart grid presentation will

be used to set context for network architecture and

design. In a traditional power grid of an electric power

system (or utility), electricity flows from bulk power

generators to consumers over a grid of transmission

lines and distribution feeders, as shown in Figure 2.A hierarchy of transmission lines is connected

through a series of transmission substations leading

to distribution substations at the edge of the grid.

(See [1] for this classification of substations.) Step-up

Transmissionsubstation





Distributionsubstation

Thermal (coal,gas), hydro-electric,nuclear

CHP—Combined heat and powerDER—Distributed energy resourcePHEV—Plug-in (hybrid) electric vehiclePV—Photovoltaic (cell)UPS—Uninterruptible power supply

Business

Residence

Residence

Largebusiness,industrialcomplex

Storage

DER

DER

PHEV

Micro-generation(PV,…)

Large scale(PV, wind,diesel, UPS,CHP, …)

Alternate,renewable

energysource

Wind,PV,

bio mass,hydro,

tidal,fuel cell,

…

(Hierarchy of)micro grids

Storage

Bulk powergeneration

Distributionsubstation

To regionalor national grid

PV

Extr

a h

igh

an

d h

igh

vo

ltag

etr

ansm

issi

on

lin

es

Med

ium

vo

ltag

e an

d(s

ub

)-tr

ansm

issi

on

lin

es

Feed

er

Bulk powergeneration

Transformer(s)

Generator

Feed

er

Alternate,renewable

energysource

Figure 2.Generation, transmission, and distribution in smart grid.


or step-down transformers at the substations are used

to change voltages to levels appropriate for the corre-

sponding transmission lines and feeders. Finally, a dis-

tribution transformer on a feeder (such as those

mounted on utility poles or underground) steps down

the voltage to the standardized level required at the

consumer locations. Some industrial and large busi-

ness customers may connect to the grid at the feeder

voltages or even at the sub-transmission voltages.

An example of voltages used in a typical power

system [7] uses generation at 15–25 kV; hierarchy of

transmission lines at 500 kV, 230 kV, and 69 kV (sub-

transmission); distribution feeders at 12kV; residential

customers at 115/230 V. Over time, the power grids of

many utilities have been interconnected to form

regional, national, and international grids improving

energy management and transmission reliability.

With the advent of cost-effective generation of

renewable and/or alternate sources of energy it is now

possible to connect these energy sources of various

capacities throughout the grid (see Figure 2). As a

result, the direction of electricity transfer will fluctu-

ate based on local weather conditions, the position of

the sun, and other environmental effects. To com-

pensate for the variable nature of photovoltaic and

wind generation sources, for example, storage ele-

ments are deployed. Some of the new storage systems

(whether associated with power generation or stand-

alone) include batteries, high-energy flywheels,

(ultra) capacitors, pumped hydro, and compressed air

energy systems. One important class of storage devices

that are expected to be prevalent in the future are

plugged-in (hybrid) electric vehicles (PEV, PHEV). In

addition to supporting transportation, when parked,

the vehicles with charged batteries can potentially be

used to supply electricity to the grid.

Smart Grid ApplicationsThe requirements of smart grid applications drive

the design and architecture of an integrated smart grid

communication network.

Traditional ApplicationsTeleprotection and supervisory control and data

acquisition (SCADA) applications have been employed

for power grid management. Teleprotection refers to

the use of signal-aided relay-to-relay communication

between adjoining substations (i.e., substations con-

nected by a transmission line). If protection equip-

ment at either end detects a fault, the other end is

notified, and protective actions such as tripping

(power circuit disconnect) are initiated in order to iso-

late the fault. SCADA systems consist of remote ter-

minal units (RTUs), programmable logic connectors,

and other intelligent electronic devices (IEDs) con-

nected over communication networks. These sensors

and actuators are located at power stations, substations,

distribution transformers, and other grid locations.

They communicate with their respective manage-

ment systems at the utility data center (centralized)

or substations (distributed). In addition to grid opera-

tions, utility communication needs may also include

support for enterprise voice and data applications.

Many utilities have deployed private land mobile

radio (LMR) networks for their mobile workforce for

group voice communications (push-to-talk) as

well as some peer-to-peer voice communication

needs.

Examples of Smart Grid ApplicationsThe NIST/EPRI roadmap documents, [5] and [17],

divide the smart grid conceptual model into seven

domains that together represent the smart grid

community of interest. These domains are bulk gen-

eration, transmission, distribution, customers (con-

sumers), grid operations, service providers (for

services such as billing and third party providers), and

markets (wholesale, retail, and trading). While some

of these domains are connected by the electric grid

(generation transmission, distribution, and cus-

tomers), all of them must communicate with each

other. The five classes of applications listed earlier are

briefly described in this section. These applications

have been chosen for their diverse network require-

ments and together they incorporate many of the net-

work architecture elements covered in this paper.

Smart metering. Smart metering is one of the first

new smart grid applications deployed by most utilities.

Smart metering encompasses much more than peri-

odic energy measurement. Many new smart grid

applications require frequent power (both active and

reactive) and power quality (e.g., voltage, frequency)


measurements. Such measurements (provided by

smart meters) may be required as often as once every

15 minutes to support energy management applica-

tions. Measurements provided by smart meters are

also used to support real time pricing (RTP), time of

use (TOU) pricing, and critical peak pricing (CPP) fea-

tures for billing and demand response applications.

1. Depending on the size of a utility, the number of

smart meters in the network can vary from a few

thousand to several million.

2. Regulatory requirements, lack of timely smart

metering standards, and cost considerations have

led to deployment of vendor-proprietary smart

metering solutions based on neighborhood area

networks (NAN). These solutions can be readily

deployed using wireless technologies deployed in

unlicensed spectrum or using power line carrier

(PLC) technologies. A meter concentrator con-

nects to the meters over the NAN and is respon-

sible for aggregating data collected from the

meters it serves. The number of meters served by

a concentrator can vary from a handful for a PLC-

based NAN to several hundred or even several

thousand for a radio frequency (RF) mesh-based

solution. The meter concentrator connects to the

meter data management system (MDMS) over an

Internet Protocol (IP) network.

3. There are smart meter products with direct inter-

faces to a wireless service or Ethernet interfaces to

connect to wireline services. For such deployment

there is no meter concentrator and the meter

communicates directly with the MDMS over an IP

network.

4. Smart meter connections to home area networks

are fundamental to residential or building energy

management. Such connections, for example,

allow appliances to respond to pricing signals or

other triggers carried over the smart grid.

5. Under normal operating conditions, the accepta-

ble response times for completing meter transac-

tions can be high. Thus, one-way packet latency

allowances can also be high—on the order of sev-

eral seconds. The availability of an individual

meter may not be considered too critical to net-

work operations; hence, an availability objective

of 99.95 percent should be reasonable (corre-

sponding to an average downtime of 263 min-

utes/year).

6. However, some smart grid applications may

require data transfer from all linked meters over

a relatively short timeframe (several minutes),

which requires low latency for each of the indi-

vidual meters, even if higher latency may be

acceptable for billing purposes.

7. While security issues are out of the scope of this

paper, it is important to note that smart meters

are, perhaps, the weakest link in smart grid secu-

rity. In addition to the security threat to electric-

ity usage data and unauthorized physical access to

the meter itself, threats attributable to wireless

connectivity (for meters thus connected) must be

considered in the architecture and design of the

network.

Automated demand response. Demand response

activity is an action taken to reduce electricity demand

in response to price, monetary incentives, or utility

directives so as to maintain reliable electric service or

avoid high electricity prices [20]. Demand response

is a temporary change in electricity consumption in

response to supply conditions or other events in the

grid [5]. The inclusion of new energy sources and

storage elements combined with the need to reduce

peak loads and conserve energy has driven the intro-

duction of distributed automated demand response

applications. ADR applications, for example, can be

used to reduce the amount of energy consumed by

appliances during peak power periods. While demand

response has been used by utilities over the years

through scheduled load shedding and manually man-

aged consumption reduction with a few large con-

sumers, ADR is much wider in scope—bringing

dynamic load management directly to residential con-

sumers. ADR often works in concert with distributed

energy resources (DER) closer to the point of con-

sumption or other energy sources connected into the

grid. Thus, in some cases, ADR may not necessarily

reduce overall energy consumption, but only transfer

the source of some of the consumed energy to DER.

Such load shifting will result in reduced carbon emis-

sions if the DER is a renewable energy resource.


Dynamic pricing mechanisms such as CPP and TOU

pricing through smart meters contribute to efficient

ADR implementation and possible cost reduction.

1. ADR is still evolving. The Demand Response

Research Center OpenADR communication speci-

fication is a data model for information exchange

between the utility and consumer facility and is

designed to automate demand response actions

at the customer location [12].

2. The latency allowance between a utility (or an

independent service operator) and a single con-

sumer’s premise should be less than one minute.

As discussed earlier, however, to span large

numbers of locations (through their respective

meters), much smaller latencies will be required

to support “sweeping” through the meters in a

“neighborhood.”

3. Electric vehicles offer another form of storage for

the smart grid, one with unique communication

networking challenges. Even with the most con-

servative estimates of PEVs and PHEVs, utilities

are ill equipped to support the increased demand

with currently deployed bulk power generation.

P(H)EVs will add tens of thousands of mobile and

roaming endpoints not only to the utility grid, but

also to communication networks. A P(H)EV can

be charged or discharged at an owner’s home,

special charging stations (e.g., at parking lots), or

other locations. Thus vehicle mobility and energy

transfer must be accurately captured through the

relationship between the vehicle communication

port and the electric port connected to the grid.

Teleprotection. The IEEE 1646 standard [8] lists

latency requirements for some of the substation opera-

tions at as little as 1/4 of a cycle (which translates to

about 4 ms and 5 ms, for 60 Hz and 50 Hz AC fre-

quencies, respectively). For applications requiring

communication between substations, the latency

requirements are relaxed to 1/2 cycle. (See Inter-

national Electrotechnical Commission [IEC] specifi-

cation 61850-5 [9] as well as [8].) Thus remote

activation of a protection scheme at a substation is

needed within 8 ms to 10 ms after a fault at that sub-

station has been remotely detected at an adjoining

substation.

Teleprotection applications require extremely

high network availability: failure of such applications

may result in destruction of grid infrastructure and,

potentially, loss of life. For this reason, utilities have

deployed redundant communication links between

substations using a variety of options including pilot

wires, leased voice grade lines, leased data lines, PLC,

and fiber including Ethernet, synchronous optical net-

work/synchronous digital hierarchy (SONET/SDH),

and microwave. To support the low latency require-

ments, connections are typically point-to-point along

the transmission line between the substations, which

is seldom longer than 300 km.

Distribution automation. Distribution automation

extends monitoring and control much deeper into the

distribution network to encompass line reclosers, volt-

age regulators, capacitor banks, sectionalizers, line

switches, fault indicators, circuit breakers, load tap

changers, and transformers. In addition to these ele-

ments, new IEDs will also need to be supported.

To date, power utilities have been accustomed to

managing a limited number of monitoring and control

points, e.g., at hundreds of substations. New commu-

nications technologies need to be introduced into their

grid operations in order to connect tens of thousands of

endpoints encountered in distribution automation in

substations and feeders. Communications with these

widely deployed endpoints can be challenging depend-

ing on the available network access mechanisms.

Micro grid management. The energy management

system (EMS) of a typical utility consists of multiple

centralized or distributed systems. The smart grid may

include and/or connect to “micro grids” possibly man-

aged by individuals or organizations. Taking the sys-

tem approach presented in [11], a micro grid with its

generation, storage, power lines, and loads becomes a

subsystem of the larger utility grid. Micro grids can

be small and simple such as within a home or may

span an interconnection of grid elements in a neigh-

borhood, or over a single feeder-based system, or over

a collection of systems connected to a distribution sub-

station (see Figure 2). In most cases, the micro grid is

an autonomous system since it may be disconnected

(involuntarily or voluntarily) from the larger grid, and

still support its consumers adequately. We assume the


existence of an EMS for the micro grid that can com-

municate with the EMS of the utility grid or other

micro grids for maintaining grid stability and to sup-

port ADR and other applications.

1. The communication network architecture should

be divided in a hierarchy consistent with micro

grids supporting local reliability if disconnected.

2. It is possible that the owner of the micro grid

communication network is distinct from the

owner of the utility network.

Overview of Smart Grid Application RequirementsTable I lists a few of the applications (or classes of

applications) and their qualitative requirements. Note

that:

1. Some of the table entries are based on [8] and [9].

2. The quantified values of the requirements depend

on the specific nature of applications and associ-

ated utility requirements.

3. “Micro grid management” refers to communica-

tion between a micro grid EMS and the EMSs in

the micro grid hierarchy.

4. Requirements for some of the applications may be

different from Table I under certain circumstances.

For example, lower latency and higher reliability

may be needed for smart metering during ADR

and emergency load management activities.

Network ArchitectureThe new grid in Figure 2 is much more than an

interconnection of transmission lines and distribution

feeders for delivering electricity to homes and busi-

nesses. Our working and descriptive definition of the

smart grid is a power grid, where its applications are man-

aged by state-of-the-art information technologies over an

integrated high-performance, reliable, and secure commu-

nication network.

Data Rate/ (One Way)Application

Scope HSData Volume Latency Reliability Securityor P2P(at Endpoint) Allowance

Smart metering HS Low/v. low High Medium High

Inter-site rapid response (e.g., P2P High/low Very low Very high Very highteleprotection)

SCADA P2P, HS Medium/low Low High High

Operations data HS Medium/low Low High High

Distribution automation HS, P2P Low/low Low High High

Distributed energy managementand control (including ADR, HS, P2P Medium/low Low High Highstorage, PEV, PHEV)

Video surveillance HS High/medium Medium High High

Mobile workforce (push-to-X) HS Low/low Low High High

Enterprise (corporate) data HS Medium/low Medium Medium Medium

Enterprise (corporate) voice P2P Low/v. low Low High Medium

Micro grid management HS, P2P High/low Low High High(between EMSs)

Table I. Qualitative comparison of application requirements.

ADR—Automated demand responseEMS—Energy management systemHS—Hub-spokeP2P—Peer-to-peerP(H)EV—Plug-in (hybrid) electric vehicleSCADA—Supervisory control and data acquisition


Figure 3 shows the composition of the integrated

communication network for a utility.

The communication network supports communi-

cation between the sensors and actuators attached to

the grid elements, and the smart grid applications

enforcing grid policies through the actuators based on

these measurements. In addition to the smart grid con-

trol applications, the integrated network supports other

utility needs such as multimedia data transfer (e.g.,

closed circuit television [CCTV]) from substations

and voice and data applications for its “enterprise” and

mobile work force. Furthermore, the network must

connect to other utilities’ smart grid networks as well as

to other entities such as the independent system opera-

tors (ISOs) and regional transmission organizations

(RTOs). Finally, for operational simplicity, it is prudent

to implement a network hierarchy consistent with the

micro grid hierarchy as shown in Figure 3.

A smart grid communication network architec-

ture is presented in this section including physical

connectivity architecture, examples of logical con-

nections, access network options, and architectural

implications of shared ownership of networks.

Physical ConnectionsFigure 4 represents the essence of networking

architecture for connecting most of the smart grid and

other endpoints, their locations, and implied applica-

tions. (Also see [17].)

As illustrated, where a network endpoint is

shown to connect to more than one network, not all

connections may be applicable in an implementation.

For example, a building may connect only to one of

four possible network connection options shown: a

renewable energy source may connect to the wire-

less or wireline access network, while a substation

may require connections to more than one network.

It is not possible to show all smart grid elements and

there will be differences from Figure 4 in connection

arrangements in an actual deployed network. For

example, a large traditional generating station in an

outlying area may connect through an access net-

work. Many control and management systems are

named generically: their implementation will depend

on the utilities and vendor products.

Despite the broad consensus that IP is a reasona-

ble choice for a smart grid communication protocol,

Electrical power network

(Hierarchy of)micro grids

Communication network

Enterprisevoice, data

Mobileworkforce

Extranet

Videosurveillance

Sensors and actuators

Figure 3.Communication network beyond smart grid control network.


NIST has stopped short of mandating IP [17]:

“Among smart grid stakeholders, there is a wide

expectation that Internet Protocol (IP)-based net-

works will serve as a key element for the smart grid

information networks. . . . An analysis needs to be

performed for each set of smart grid requirements to

determine whether IP is appropriate.” We do assume

here that IP is the networking protocol of choice for

the integrated network. Additionally, for many utili-

ties it may be prudent to implement Multiprotocol

Label Switching (MPLS) virtual private networks

(VPNs), with each VPN supporting a set of applications or

user communities. In Figure 4, direct connections

to the core network are assumed to be over point-

to-point (layer 1 or layer 2) connections. There may

be more than one data and control center for relia-

bility and load sharing of the systems. It is expected

that many smart grid applications will require dis-

tributed control and management systems, which

could be located at the substations and in the corre-

sponding micro grids. The network must also con-

nect to the networks of other utilities in the

regional/national smart grid. Also (extranet) con-

nectivity is required to the utility’s partner ISO/RTOs

and corporate service providers for billing, installa-

tion, and other services.

CCTV—Closed circuit TVDER—Distributed energy resourceEMS—Energy management systemHAN—Home area networkIP—Internet Protocol

Power station(large, traditional)

Distributed alternate renewable power generation(e.g., PV, wind, bio mass, tidal, microturbines)

Meter dataman. sys.

Voice/data/push-to-xsystems

SCADAman. sys. …

Utility data and control center

ExtranetconnectivityEMS

RTO/ISO

Wir

eles

s ac

cess

net

wo

rk

Wir

elin

e ac

cess

net

wo

rk

Neighborhood area network

Power line communication network

HAN/(enterprise) LAN

Meter

Building (residential, business, industrial, other)

Distributed energyresources

Utility “pole”

Vehicle(PEV, PHEV)

SCADAMeter

concentrator

Essential

Storage

Vehicle chargingstation

Mobileworkforce

Distributionman. sys.

Videosurveillance

if present if present if present

Billingsystem

Voice/data

Utility office

Micro gridEMS

Storage

Substation

SCADAMeter

concentrator…Protection CCTV

Voicedata

SCADAman. sys.

EMS

if present

EMS

(IP/MPLS) core network

PV

PMU

WAMSman. sys.

PMU—Phasor measurement unitPV—PhotovoltaicRTO—Regional transmission organizationSCADA—Supervisory control and data acquisitionWAMS—Wide area monitoring system

ISO—Independent system operatorLAN—Local area networkMan. sys.—Management systemMPLS—Multiprotocol Label SwitchingPE(H)V—Plug-in (hybrid) electric vehicle

Figure 4.Physical connectivity architecture.


Every consumer location is expected to have a

smart meter that is connected to the network. A con-

sumer building may have DERs, energy storage ele-

ments, and/or electric vehicles. DERs may be classified

as microgeneration (at residences and small busi-

nesses) or large-scale (at large business or industrial com-

plexes). Depending on their engineered capacity, DERs

provide between 3kW and 10,000kW power. DERs are

typically located in or close to a consumer building

[4]. When present in consumers’ premises, DER, stor-

age, vehicles, and meters may connect over a HAN

or a LAN, which is in turn linked to the communica-

tion network—often through the meter for residential

consumers.

The grid equipment at distribution points

(e.g., utility poles) connects to the network over one

or more of the connections shown in Figure 4.

Standalone (and large scale) alternate/non-traditional/

renewable energy sources, storage elements, and

vehicle charging stations connect to the wireless

and/or wireline access networks. Each micro grid has

its own communication network similar (but smaller

in scale) to that shown in Figure 4. The micro grid

EMS connects to the utility or other micro grid net-

work.

It is expected that Voice over Internet Protocol

(VoIP) will be used for peer-to-peer (P2P) voice and

push-to-talk (PTT) communication of the mobile

work force. It is also expected that mobile wireless

data applications will migrate to broadband. Thus, all

voice, push-to-x, data, and video needs can be satis-

fied by the mobile access terminal connecting to the

wireless broadband access network. Until VoIP com-

munication is available, gateways will be needed to

connect legacy voice systems to the smart grid com-

munication network.

Depending on its location and size, a substation

may house many smart grid and other systems that

require communication with other endpoints. Only

a few of these systems are shown in Figure 4. Figure 5is a more detailed schematic of a substation’s com-

munications infrastructure and its connectivity to vari-

ous possible wide area networks (WANs). This

illustration depicts only the generic systems requir-

ing communication with the outside world.

The IEC 61850 standard provides comprehensive

specifications for substation automation for connect-

ing substation systems that support grid operations.

These systems include legacy analog and digital sys-

tems as well as new IEDs (e.g., for SCADA and pro-

tection). The resulting substation automation LAN

may be implemented as a hierarchy of Ethernet LANs

where a station bus connects many process busses

with each process bus connected to multiple modern

and legacy substation systems. For a typical substa-

tion LAN see [2].

The substation may have a separate LAN for

applications that do not directly contribute to automa-

tion. The substation router may need to connect to

more than one network from the set of networks

shown in Figure 5. Depending on the network tech-

nology, network access adapters or gateways will be

needed. An application such as teleprotection may

need network connections to an adjoining substation

in addition to or instead of the connections through

the substation router. Over time, different networking

technologies have been used with network-specific

gateways facilitating such connections. IEC is working

on extending the 61850 standard to support commu-

nication between substations using multicast over

Ethernet, which would allow substation automation

LANs to connect over an Ethernet network. Further,

tunneling Ethernet over an IP connection through

the router is a possibility if latency objectives can be

met. (See [21] for an overview of teleprotection con-

nectivity options, including Ethernet.)

Logical Connection ModelsIrrespective of their physical network connec-

tions, it is important to determine endpoints of an

application carried over the network. As an illustra-

tion, logical connection models for a few of the appli-

cations are shown in Figure 6, highlighting the

interdependence of the applications—particularly

among ADR, SCADA management, and distribution

management.

Depending on the grid elements under its con-

trol, an EMS communicates only with a subset of the

generation and storage elements shown in Figure 6.

The smart meter that plays a central role in many

applications has been replicated (as entity M) for ease


of presentation. Newer distribution automation appli-

cations such as the Volt, VAR, Watt control (VVWC)

application will increasingly use periodic as well as

on-demand smart meter measurements. Additionally,

the smart meter is a centerpiece of ADR applications

and home energy management. (Also see [5].)

Networking TechnologiesThis section presents networking technologies for

the network segments shown in Figure 4.

PLC networks. Communication over power line

carriers in various bands between 10 Hz and 500 KHz

have been used by utilities for some of their commu-

nication needs. The main advantage of PLC is the

availability of power lines connecting every endpoint

that needs to be connected to the communication net-

work. However, PLC shortcomings include low data

rates, interference over long distance high voltage

lines, and the need to connect PLCs across trans-

formers (except when the PLC carrier frequency is

identical to the 50 Hz or 60 Hz line frequency.) While

traditionally data rates have varied from about 15 bits

per second for line frequency carriers to about 3 to 4

kilobits per second (kb/s), new orthogonal frequency

division multiplexing (OFDM)-based techniques will

allow for rates as high as 130 kb/s [15]. The current

draft of the emerging IEEE P1901 Broadband Over

Power Line (BPL) standard indicates future support

for much higher rates, i.e., in the tens of megabits per

second range.

Neighborhood area networks. PLC is one example

of NAN connecting buildings in a neighborhood.

Another example is RF mesh over unlicensed spec-

trum, such as over the 900 MHz industrial, scientific,

and medicine (ISM) band or the 2.4 GHz band. While

NANs are predominantly used for smart metering,

One or more network-specific adapters, or gateways

Transmissionline PLC

Feeder PLC NANWireless

access networkWireline

access network

Corenetwork

Substation

Router

Ad

dit

ion

al n

etw

ork

(s)

for

tele

pro

tect

ion

Meterconcentrator CCTV

ADRmanagement

system

Data

Voice…

ADR—Automated demand responseCCTV—Closed circuit TVEMS—Energy management systemLAN—Local area network

EMSPMUProtectionSCADA

managementsystem

SCADA

Ethernet,optical,PDH, SDH/SONETPLC, other

Substation automation LAN(of station bus and process bus)

…

“Gat

eway

”

SCADA—Supervisory control and data acquisitionSDH—Synchronous digital hierarchySONET—Synchronous optical network

NAN—Neighborhood area networkPDH—Plesiochronous digital hierarchyPLC—Power line carrierPMU—Phasor measurement unit

Figure 5.Substation network.


other applications may also be carried such as SCADA

RTUs and IEDs located at the neighborhood trans-

formers. Lately Wi-Fi has also become a viable NAN

technology for these applications.

Wireless access. Wireless broadband access is one

of the catalysts for successful deployment of the smart

grid. In many countries, with the exception of unli-

censed spectrum, most of the spectrum is owned by

service providers and available for broadband services

over technologies such as Code Division Multiple

Access 2000 (CDMA2000), Enhanced Data Rates for

GSM Evolution (EDGE), Universal Mobile Telecom-

munications System (UMTS), High Speed Packet

Access (HSPA), Worldwide Interoperability for Micro-

wave Access (WiMAX), and Long Term Evolution

(LTE). Depending on the technology and configura-

tion, data rates between 500 Kb/s and 3 Mb/s or more

are possible (1 Mb/s to 6 Mb/s or more on downlinks).

With little spectrum available for their exclusive use,

utilities may subscribe to wireless broadband services

Meter

Meter

Storage Meter

Buildingor other energy

consumption entity

Storage

Power station(large,

traditional)

Distributed alternate renewablepower generation

(e.g., solar, wind, bio mass,tidal, microturbines)

Meter dataman. system

SCADAman. sys.

(Utility) EMS

RTO/ISO

SCADA(sensor)

Micro gridEMS

Electric vehiclechargingstation

Distributionman. sys.

Billingsystem

Measurements(periodic,

on-demand)

Measurements(periodic)

Power qualitymeasurements

(periodic,on-demand)

Measurements(periodic,

on-demand)

On-demand

Regulate

SCADA systems

RegulateStatus,

measurements,incidents

Releaseenergy

Releaseenergy

Increase,decrease

shut-down

Increase,decrease,

shut-down

Increase,decreaseenergy

Demand

Storeenergy

Storeenergy

Demand

Demand

DERElectricvehicle

Building

Micro grid

ADRman. sys.

M

M

M

Homeappliance

Schedule,stop

ADR—Automated demand responseCCTV—Closed circuit televisionDER—Distributed energy resourceEMS—Energy management systemISO—Independent system operatorMan. sys.—Management system

SCADA(actuator)

Regulate

PMU

WAMSman. sys.

(Processed)measurements

Measurements

To otherutilities,

control centers,etc.

Demand

Increase,decrease

Pricing(policies)

Pricingsignals

Pricing signals,periodic poll,on-demand

P(H)EV—Plug-in (hybrid) electric vehiclePMU—Phasor measurement unitRTO—Regional transmission organizationSCADA—Supervisory control and data acquisitionWAMS—Wide area measurement system

On-demand

Measurements

Status

Figure 6.Logical connection models for a few smart grid applications.


if service providers offer the required coverage, satisfy

the security and reliability requirements, and support

preferential treatment for critical utility applications

when needed. As an example, utilities in the United

States have dedicated narrowband spectrum, essen-

tially to carry voice services over their private mobile

radio network, with little to no prospect for future

broadband allocations. In some cases, utilities may

be able to acquire spectrum for their smart grid net-

work if such spectrum is available with the corre-

sponding product support. The recent allocation of 30

MHz of spectrum in the 1.8 GHz band to electric utili-

ties in Canada is a prime example where spectrum was

indeed assigned but in a band not considered by wire-

less standardization bodies at this time, thus requiring

product customization.

While spectrum can be made available in a few

countries, one possibility is for utilities to enter into a

partnership with wireless service providers or a spec-

trum licensee, for sharing resources such as spectrum,

towers, equipment cabinets, and/or even network

equipment with acceptable logical partitioning. The

relationship between a utility and a wireless service

provider, whether a partnership for resource sharing

or direct customer-provider access agreements, will

have the corresponding impact on network architec-

ture and design.

Wireline access. Broadband wireline services like

digital subscriber line (DSL), cable, Gigabit passive opti-

cal network (GPON), and BPL can provide the band-

width needed for smart meters, SCADA equipment,

and meter concentrators located at the distribution

transformers, or aggregation of substation-based appli-

cations traffic through substation routers. Some utili-

ties may be averse to allow residential broadband

connections to be used for smart meter traffic since

the connection is shared among other applications

of the homeowner. On the other hand, if the broad-

band service and the utility are owned by the local

government of a (small or medium sized) commu-

nity, the residential broadband connection may in fact

be preferred by the utility. (Note that, in spite of its

name—broadband over power line—BPL uses the

power line only between the secondary of the distri-

bution transformer and home. The BPL connection

between the utility pole and the central office is often

a fiber connection).

As noted before, direct SDH/SONET and direct

Ethernet networks are also used for connecting adjoin-

ing substations for applications like teleprotection.

Core network. Depending on the number of com-

munication endpoints of a utility, their locations, their

communication requirements, and other require-

ments, a core network may consist of a router at the

utility data center, or routers connected over an opti-

cal ring or metro-Ethernet in a metropolitan area, or

a mesh of routers connected over point-to-point links.

For the general mesh, the point-to-point links are

either utility-owned or leased from layer 1 (L1) (pri-

vate lines) and L2 (e.g., Ethernet, frame relay) ser-

vice providers. If the performance, reliability, and

security requirements are acceptable, the core net-

work can also be an MPLS VPN from an L3 service

provider with the utility routers connecting to the

provider edge routers of the MPLS VPN service [16].

The utility may itself provide MPLS VPN service so

that groups of users, applications, and/or locations can

manage their networking needs in an autonomous

fashion. In a case where the core network itself is an

MPLS VPN over a service provider network, the VPNs

within the utility can still be created independent of

and within the service provider VPN.

Network Ownership: Utility-Owned Versus Public Carriers

Most utilities prefer exclusive end-to-end owner-

ship of the network though this may not always be

possible because of cost considerations, spectrum availa-

bility, the need for deploying applications in an expe-

dient manner, and other considerations. There are

many advantages and a few serious drawbacks

(e.g., costs) for utility-owned networks over shared own-

ership of network segments with service providers. The

fact that multiple parties share ownership of the net-

work segments within an integrated network does

affect the network architecture’s physical and logical

connectivity, routing, reliability, security, and other

factors. In addition to commercial and business con-

siderations, interoperability agreements between the

utility and network service providers can influence


the network architecture in its end-to-end network

operations, monitoring, and management challenges.

Even if the utility owns the end-to-end network,

the point-to-point L1 and L2 connections may still be

leased from service provides with acceptable capacity,

reliability, security, and preferably with exclusivity.

However, implementing exclusivity on L1 and L2

links over a wireless service provider network sup-

porting utility requirements for their mission critical

applications can be challenging.

Interconnection With Network of SynchrophasorsWidespread blackouts at the beginning of this cen-

tury have underscored the necessity of wide area mea-

surement systems (WAMS) for regional or national

grids across all the connected utilities’ power systems.

WAMS employ phasor measurement units (PMUs) to

measure voltage and current phasors (phase vectors)

of the corresponding alternating current waveform

and its harmonics. PMUs are considered to be the state

of the art SCADA RTU and are expected to be

deployed at a large number of locations in participat-

ing utility grids. PMUs (often called synchrophasors)

are synchronized to a common clock, usually derived

from the Global Positioning System (GPS). This allows

for time-stamped measurements shared among

utilities, regulatory bodies, and other organizations

through PMU gateways connected to a regional or

national network such as the one being developed and

deployed by the North America SynchroPhasor

Initiative (NASPI) [14]. The NASPI network (NASPInet)

will be a high performance, reliable, and secure com-

munication network that connects PMU gateways

among utilities in a region to a distributed data bus,

allowing for PMU data sharing (almost instantaneously

for some applications) between utilities as well as orga-

nizations such as the North America Electric Reliability

Corporation (NERC).

Therefore, a utility smart grid network will need

to connect to (and be a part of) a network like

NASPInet.

Network Design Principles to Facilitate Smart Grid Applications

This section deals with topology, QoS, and relia-

bility considerations that may differ from the time-

honored design methodologies for service provider

networks and even many large enterprise networks. A

few implementation and product development chal-

lenges also will be identified, with suggestions for

workarounds where possible.

Network TopologyFrom the earlier discussions on smart grid, utility

applications, and the placement of application end-

points, the obvious choice for the network topology is

predominantly a tree structure. The initial topology

of an iterative design will be similar to the one shown

in Figure 7 before QoS, reliability, and other require-

ments are applied to determine the final topology

design.

In addition to the centralized destination of the

utility data and control center for a significant amount

of applications traffic, a substation router is perhaps

the other most identifiable location of traffic aggre-

gation as shown in Figure 5. Traffic from multiple

(smaller) substations may be aggregated at another

(large) substation. Depending on the meter technolo-

gies deployed, the metering traffic may be aggregated

at the meter concentrators at substations, or the

meters or concentrators may connect directly to

the core network. Finally, other smart grid endpoints

such as energy sources and storage units connect to

the substations or directly to the core network routers,

depending on their locations and/or the location of

the corresponding EMSs.

In principle, peer-to-peer applications can be sup-

ported over the tree topology, since connectivity is

always possible through the core routers. However,

the latency requirements of some applications may

not be satisfied by routing that traffic through the core

network. Many of these low-latency applications have

endpoints in adjoining substations. Thus, it is prudent

to maintain direct communication link(s) between

substations as shown in Figure 6 and described fol-

lowing Figure 5. These direct inter-substation links

may additionally provide the possibility of a preferred

path for other P2P traffic such as VoIP bearer.

The topology design is also affected by the fact

that the choice of access network is driven more

by the coverage of a large number of endpoints than


by the traffic volume. Further, for most centralized

applications, the upstream traffic volume is greater

than the downstream traffic, requiring special design

considerations when carried over service provider

networks that are generally optimized for higher

downstream traffic than upstream traffic.

Integrating Legacy Applications and NetworksWhile new applications and new smart grid sys-

tems are expected to support IP connectivity, the inte-

grated smart grid communication network will also

have to connect to applications at legacy systems for

a period of time. In most cases, gateways to the legacy

systems will be required. Depending on the evolution

of end systems, these gateways can be as simple as

those providing serial-to-Ethernet conversion to those

supporting full application layer protocol conversion.

For a few analog applications including push-to-talk

voice, circuit emulation (time division multiplexing

[TDM] over Ethernet or IP) will have to be provided

until these applications migrate to IP (e.g., VoIP).

Quality of ServiceThe two important QoS factors considered here are

a wide range of latency requirements and (dynamic)

association of flow priority to applications consistent

with smart grid operations. With smaller data vol-

umes, efficiency in bandwidth allocation to applica-

tions may not be the most important QoS objective in

smart grid network design.

Managing latencies. Throughout this paper, the

need for supporting applications with diverse latencies

(from about 8 ms to 1 second or more) was empha-

sized. But lower latency does not always imply higher

(IP/MPLS) core network

Utility data and control center

Substation

SubstationSubstationSubstation

Building BuildingBuilding

SCADA

Meterconcentrator

Mobileworkforce

Micro gridEMS

Micro gridEMS

RTO/ISOPower station

(large,traditional)

Distributed alternate renewablepower generation (including DER)


Storage

Mobileworkforce

Distributed alternate renewablepower generation (including DER)

Utility office


Substation

DER—Distributed energy resourceEMS—Energy management systemIP—Internet ProtocolISO—Independent system operator

Storage

MPLS—Multiprotocol Label SwitchingRTO—Regional transmission organizationRTU—Remote terminal unitSCADA—Supervisory control and data acquisition

Meter Meter Meter

Figure 7.Network topology.


priority. For example, according to Internet Engineering

Task Force (IETF) Request for Comments (RFC) 4594

guidelines [3], network control traffic with low

latency (delay tolerance) characterization is given

higher priority with a differential services (DiffServ)

class selector 6 (CS6) while the VoIP bearer traffic

with very low latency is allocated to a lower priority

expedited forwarding (EF) class. In most data net-

working implementations, as in RFC 4594, the VoIP

bearer traffic is given a higher priority than any class

of applications (other than network control), often

assigning it to the strict priority egress queue at each

router. The primary reason for such a high priority is

to manage the jitter and packet loss of the bearer traf-

fic even if there may be real-time or business appli-

cation traffic that requires higher priority. Managing a

diverse set of application priority and latency require-

ments for a smart grid network will require a different

QoS design approach. The QoS design should begin

with listing all utility applications and their priority

and latency requirements. Table II is a sample of

utility applications with their priority and latency

requirements.

Using VoIP bearer traffic as a reference, applica-

tions such as teleprotection and PMU data transfer to

NASPInet have much lower latency requirements

than the latency allowance of up to 175 ms to 200 ms

necessary for good voice quality under most condi-

tions, yet voice traffic latency is considered very low

in RFC 4594. As can be seen from Table II, there are

smart grid applications with much lower latency

requirements and very high priority. The only plausi-

ble design choice for satisfying 8 ms to 16 ms latency

requirements may be in directly connecting the appli-

cation endpoints such as two substations, thus elimi-

nating intermediate hops and reducing propagation

LatencyApplication (only a few allowanceexample applications (assumed,

considered) Application setting unverified) Comments

Teleprotection All 8 ms, 10 ms For 60 Hz and 50 Hz, respectively

Phase measurement unit Class A data service 16 ms60 messages per second stipulated for Class A data service in [14]

Push-to-talk signaling Incident-related 100 ms

Connect to many Example: ADR within 1 minute for up Smart meter meters in a short 200 ms to 300 meters connected over a shared

time medium

SCADA data: poll response 200 ms See [8].

VoIP bearer 175–200 ms Includes P2P and all PTT

VoIP signaling 200 ms Includes non-incident-related PTT

Phase measurement unit Class C data service 500 msPost event (latency value assumed).See [14].

On demand SCADA 1 second See [8].

Smart meterPeriodic meter

� 1 secondSay, once an hour or lower frequency

reading of reading

Table II. (Representative) latency requirements of smart grid applications.

ADR—Automated demand responseP2P—Peer-to-peerPTT—Push-to-talkSCADA—Supervisory control and data acquisitionVoIP—Voice over IP

In t

he

ord

er o

f d

ecre

asin

g p

rio

rity

c


delay (also see Figure 7). Packet loss and jitter con-

siderations for some other applications (e.g., VoIP)

carried over this inter-substation connection may

have to take a back seat, with lower priority. (But

note that many new codecs do correct for packet loss

and jitter.) Even though RFC 4594 lists many appli-

cation classes that may be implemented with judi-

cious choices of DiffServ values, for many practical

QoS implementations in service providers or enter-

prise data networks supporting typical multimedia

applications, four or fewer classes of service are usu-

ally provided. It is up to the network customers to

map their applications to the pre-defined QoS classes.

One such typical classification is shown in Figure 8,

alongside a more granular smart grid application pri-

ority hierarchy that is similar to Table II.

Thus with DiffServ QoS, differential services code

point (DSCP) allocation to smart grid applications will

have to be different from RFC 4594 guidelines. In

addition, while preemption of a packet under trans-

mission (either one partially transmitted or transferred

to a very small line buffer just before transmission) is

not allowed in most products or network implemen-

tations, critical high priority smart grid applications

may require the preemption feature. The use of inte-

grated services (IntServ) QoS will have to be consid-

ered for some of the applications. Since existing

networking products may not support some of these

Dec

reas

ing

pri

ori

ty

Network controlNetwork control

Teleprotection

PMU (class A data service)

PTT signaling (incident-related)

Smart metering(access many meters in a short time)

SCADA (poll response)

VoIP bearer (including PTT)

VoIP signaling (including some PTT)

PMU (class C data service)

On demand SCADA

Smart metering(periodic meter reading)

Critical enterprise/operation data

Non-criticalenterprise/operations data

Active ADR

Best effort data

VoIP bearer

Critical data

VoIP signaling

Video Video

Non-critical data

Best effort data

Smart grid application priorities Typical multimedia networkapplication priorities and QoS classes

Class 1

Class 4

Class 3

Class 2

ADR—Automated demand responsePMU—Phasor measurement unitPTT—Push-to-talk

QoS—Quality of serviceSCADA—Supervisory control and data acquisitionVoIP—Voice over IP

Figure 8.Smart grid application priorities.


required features, clever workarounds will be required;

using per flow QoS is one such possibility. Future net-

working product architects must seriously consider

developing features that support the smart grid com-

munication networks’ QoS requirements.

Finally, the importance of respecting latency

requirements of some of the ADR applications per-

taining to renewable variable energy sources such as

wind and PV cannot be overstated. Longer transients

due to a lack of efficient QoS design can lead to per-

turbation in the power grid.

Managing dynamic flow priority. Latency tolerance

and/or priority for smart grid applications may depend

on the context or setting of the corresponding grid

operation or environment. For example (see Table II

and Figure 8), periodic meter reading traffic may be

given lower priority and liberal delay allowance,

whereas traffic from meters located in an area with

active demand response processes or outage manage-

ment must be given higher priority and a lower delay

allowance. Setting a higher priority than normal to

PTT signaling and bearer traffic during an emergency

and widespread blackout or to video surveillance traf-

fic after detection of an incident are other examples.

One possible workaround is to treat an application

with varying latency and priority requirements as

multiple distinct applications. If DSCP is used, depend-

ing on the application setting, different DSCP value

should be assigned to the same application. Since

workarounds may not always be possible, standards

and product capabilities are needed for a generalized

mapping of the tuple �application, priority� to a QoS

class.

ReliabilityIt is extremely important that the (smart) grid

reliability requirements are translated into consistent

communication reliability requirements.

As noted earlier, communication network relia-

bility objectives for different applications can be very

different: 99.95 percent availability (with an average

downtime of 263 minutes/year) may be sufficient for

periodic meter reading, but 99.999 percent availabil-

ity (with an average downtime of 5.3 minutes/year)

may be low for teleprotection. To support the latter

requirement, multiple point-to-point connections

between a pair of adjoining substations are essential.

Depending on the connection option, availability of

the corresponding IP network elements, regulatory

requirements and standards, and/or a utility’s prefer-

ence, one or more of these inter-substation connec-

tions may not be included in the integrated IP

network design. However, every effort must be made

to include at least one of these links in the IP network

(e.g., Ethernet connection [21]). In addition to incor-

porating the teleprotection application in the

integrated network, such interconnections between

substations provide increased reliability for all appli-

cations carried over that connection. If needed, a judi-

cious design of the routing protocol and/or MPLS VPN

implementation will help limit the use of these mul-

tiple links to applications with high reliability require-

ments and for peer-to-peer applications.

Conventional but critical reliability design ele-

ments including substation communication link diver-

sity, redundant meter concentrators, and disaster

recovery plans for data centers and other establish-

ments must be considered to achieve the required net-

work availability goals. Since the communication

network is used for managing the power grid, it

is imperative that the network elements are not

impaired by power outages. At a minimum, battery

backups or an uninterruptible power supply (UPS) is

necessary for communication systems as well as for

some end systems. Further, PLC cannot be the sole

means of connectivity for many applications.

Green BenefitsThe introduction of smart grid applications does

contribute to green benefits as highlighted in the

Smart2020 report [10], which predicts a 14 percent

reduction by year 2020 for global carbon emission

attributable to smart grid evolution, corresponding to

a reduction of 2.03 GtCO2e, from the current emis-

sion of 14.26 GtCO2e. It is believed that 24 percent

of the total carbon emission today is attributed to the

power sector. Accordingly, the expected reduction in

overall carbon emission due to the electric power sec-

tor should be about 3 percent. Of course, this is pos-

sible partly due to the use of renewable and alternate

energy sources, peak power reduction, and energy


sources closer to load. The last item mostly refers to

reduction in transmission (I2R) losses, since a signifi-

cant amount of the electricity carried over transmis-

sion lines is generated by fossil fuels today.

An integrated communication network helps

make a smart grid more efficient, thus indirectly con-

tributing to green benefits.

There are also some green benefits, however

small, that are directly related to the time saved in

automated policy execution through communication

networks. For example, automated demand response

with effective communication can reduce the amount

of time needed for manually shifting from bulk elec-

tric sources to DER. For illustration, if it takes 20 min-

utes for manual shifting of an energy source, for every

kW of power shifted using ADR, 0.333 kWh less

energy will be drawn from the bulk electricity source

than with manual operation. Under the assumption

that the bulk electricity source is coal, a 40 percent

thermally efficient power plant could produce an

average of about 0.83 kg of CO2 emissions per kWh of

generated electricity [19]. Further, if the resulting

DER used with ADR is a renewable source of energy,

our assumption of a 20 minute savings in ADR opera-

tion would yield a reduction of 0.277 kg in CO2 emis-

sion for every kW of power shifted. Assuming that

such a shift of energy sources occurs once every day,

the average annual carbon savings achieved by the

use of ADR over a manual demand response opera-

tion is about 100 kg of CO2 for every kW of power

shifted.

ConclusionsThis paper presents a network architecture for an

integrated high performance and highly reliable com-

munications network for the successful deployment

and operation of a smart grid. The architecture frame-

work was driven by the smart grid applications—

mission-critical and otherwise—as well as other utility

applications that must be carried over the integrated

network that meet or exceed their individual require-

ments. Throughout the paper, a few representative

smart grid applications were used to illustrate the net-

work architecture. These applications ranged from

smart metering with a very large number of endpoints,

to teleprotection with extremely low latency require-

ments, to communication between autonomous

micro grid networks. Network security (including

cyber security), an extremely important aspect of

network architecture, was not considered in this

paper.

It is clear that communication network design for

the smart grid requires network topology, QoS, and

reliability considerations that may not be common-

place in designing service provider or enterprise data

networks. While it may not be possible to implement

the optimal design with available product and net-

work technologies, workarounds may be used.

Finally, the “green benefits” of the smart grid—

and by implication, that of the integrated communi-

cation network—were presented in terms of carbon

reduction.

Recommendation for Future WorkWe believe that the correlation between the smart

grid architecture and the corresponding physical and

logical connectivity of the network architecture must

be exploited in developing the smart grid architec-

ture. This holistic view of the smart grid and its com-

munication network will facilitate an easier extraction

of the network architecture from the smart grid archi-

tecture, including direct connection between their

respective performance, reliability, and security re-

quirements. Even if such “greenfield” smart grid

implementations may not be practical in most

instances, ongoing development of smart grid archi-

tecture and design, as well as new grid applications,

can facilitate the corresponding development in net-

work architecture and design. For example,

1. Tools that help determine network configurations

as an integral part of new application develop-

ment and deployment.

2. Network protocols that help reduce power tran-

sients, particularly those attributable to variable

energy resources connected into the grid.

3. Automatic setting of QoS configurations when

application requirements change based on grid

events.

4. Translating the self-healing grid to the self-healing

communication network.


Additional recommendations for future work:

1. QoS management of applications traffic with a

large variety of performance requirements includ-

ing latency and priority.

2. Extending the smart grid architecture to specific

micro grids such as a micro grid spanning a build-

ing, a feeder, and an electric vehicle charging

station.

3. Extending the architecture and design principles

introduced in this paper to include network secu-

rity. It is important to note that security consid-

erations must be incorporated at the beginning of

network architecture and design process.

AcknowledgementsWe want to thank Marc Benowitz and Sam

Samuel, co-editors of this special issue, and an anony-

mous reviewer for their review and valuable sugges-

tions. We also thank Joe Morabito for his comments

on an early draft of the paper.

*TrademarkIntelligrid is a registered trademark of Electric Power

Research Institute, Inc.

References[1] R. Albert, I. Albert, and G. L. Nakarado,

“Structural Vulnerability of the North AmericanPower Grid,” arXiv:cond-mat/ 0401084v, Jan.2004, �http://arxiv.org/ PS_cache/cond-mat/pdf/0401/0401084.pdf�.

[2] L. Andersson, C. Brunner, and F. Engler,“Substation Automation Based on IEC 61850with New Process-Close Technologies,” Proc.IEEE Bologna PowerTech Conf. (PowerTech‘03) (Bologna, It., 2003), vol. 2.

[3] J. Babiarz, K. Chan, and F. Baker,“Configuration Guidelines for DiffServ ServiceClasses,” IETF RFC 4594, Aug. 2006,�http://www.ietf.org/rfc/rfc4594.txt�.

[4] California, The California Energy Commission,“California Distributed Energy ResourcesGuide,” 2009, �http://www.energy.ca.gov/distgen/index.html�.

[5] Electric Power Research Institute, Report toNIST on the Smart Grid InteroperabilityStandards Roadmap, Contract No. SB1341-09-CN-0031—Deliverable 7, June 17, 2009,�http://nist.gov/smartgrid/InterimSmartGridRoadmapNISTRestructure.pdf�.

[6] Electric Power Research Institute andElectricity Innovation Institute, IntelliGridArchitecture Report: Volume 1, IntelliGridUser Guidelines and Recommendations,1012160, Final Report, 2002.

[7] L. L. Grigsby (ed.), Electric Power Generation,Transmission, and Distribution, 2nd ed., CRCPress, Boca Raton, FL, 2007.

[8] Institute of Electrical and ElectronicsEngineers, “IEEE Standard CommunicationDelivery Time Performance Requirements forElectric Power Substation Automation,” IEEE1646–2004, Feb. 25, 2005.

[9] International Electrotechnical Commission,“Communication Networks and Systems inSubstations,” IEC 61850-1–61850-10, 2003.

[10] C. Kruse and B. Singanayagam, “SMART2020:ICT and a Low-Carbon Economy,” J. P.Morgan, Europe Equity Research, July 9,2008.

[11] R. H. Lasseter and P. Paigi, “Microgrid: AConceptual Solution,” Proc. 35th Annual IEEEPower Electronics Specialists Conf. (PESC ‘04)(Aachen, Ger., 2004), vol. 6, pp. 4285–4290.

[12] Lawrence Berkeley National LaboratoryDemand Response Research Center, OpenAutomated Demand ResponseCommunications Specification (Version 1.0),CEC-500-2009-063, California EnergyCommission, Public Interest Energy ResearchProgram, Apr. 2009, �http://drrc.lbl.gov/openadr/pdf/cec-500-2009-063.pdf�.

[13] Michigan, Department of Labor and EconomicGrowth, “21st Century Energy Plan,”Alternative Technologies Workgroup, MeetingHandout 1, July 27, 2006, �www.dleg.state.mi.us/mpsc/electric/capacity/energyplan/alttech/smartgrid_draftreportoutlinejul19_2006.pdf�.

[14] National Energy Technology Laboratory,“Specification for North AmericanSynchroPhasor Initiative (NASPI),”Attachment A, Statement of Work, May 2008,�http://www.naspi.org/ resources/dnmtt/naspinet/quanta_sow.pdf�.

[15] PRIME Project, “PHY, MAC and ConvergenceLayers,” White Paper, v1.0, 2008, �http://www.iberdrola.es/webibd/gc/prod/en/doc/MAC_Spec_white_paper_1_0_080721.pdf�.

[16] E. Rosen and Y. Rekhter, “BGP/MPLS IPVirtual Private Networks (VPNs),” IETF RFC4364, Feb. 2006, �http://www.ietf.org/rfc/rfc4364.txt�.


[17] United States, Department of Commerce,National Institute of Standards andTechnology, Office of the National Coordinatorfor Smart Grid Interoperability, NIST SpecialPublication 1108, NIST Framework andRoadmap for Smart Grid InteroperabilityStandards, Release 1.0, Jan. 2010,�http://www.nist.gov/public_affairs/releases/smartgrid_interoperability_final.pdf�.

[18] United States, Department of Energy, EnergyInformation Administration, “InternationalEnergy Statistics, Total Electricity NetConsumption (Billion Kilowatthours),”�http://tonto.eia.doe.gov/cfapps/ipdbproject/iedindex3.cfm?tid�2&pid�2&aid�2&cid�&syid�1980&eyid�2007&unit�BKWH�.

[19] United States, Department of Energy, EnergyInformation Administration, “VoluntaryReporting of Greenhouse Gases Program—Fuel and Energy Source Codes and EmissionCoefficients,” �http://www.eia.doe.gov/oiaf/1605/coefficients.html�.

[20] United States, Department of Energy, FederalEnergy Regulatory Commission, 2007Assessment of Demand Response andAdvanced Metering, Staff Report, Sept. 2007,�http://www.ferc.gov/legal/staff-reports/09-07-demand-response.pdf�.

[21] S. Ward, W. Higinbotham, E. Duvelson, and A. Saciragic, “Inside the Cloud—NetworkCommunications Basics for the RelayEngineer,” Proc. 61st Annual Conf. forProtective Relay Engineers (CPRE ‘08)(College Station, TX, 2008), pp. 273–303.

(Manuscript approved March 2010)

KENNETH C. BUDKA is a senior director of the Network and Performance Reliability Department atAlcatel-Lucent Bell Labs in Murray Hill, New Jersey. He received a B.S. degree(summa cum laude) in electrical engineeringfrom Union College in Schenectady, New

York, and M.S. and Ph.D. degrees in engineering sciencefrom Harvard University in Cambridge, Massachusetts.Dr. Budka’s professional interests are in the devel-opment of next-generation wireless and wirelinecommunication technologies and their application tomission-critical communications systems for publicsafety agencies and utilities. He is a senior member ofthe IEEE and a former distinguished member oftechnical staff at Bell Labs. He holds 18 U.S. patents

JAYANT G. DESHPANDE is a member of technical staff in the Network Performance and ReliabilityDepartment at Alcatel-Lucent Bell Labs inMurray Hill, New Jersey. He holds a B.E.degree from Nagpur University, India;master’s degrees from the Indian Institute

of Technology, Kanpur, India, and Princeton University,Princeton, New Jersey; and a Ph.D. from the Universityof Texas at Austin. His professional interests are insmart grid architecture, design, and performance. Hehas spent the last 26 years at Alcatel-Lucent Bell Labsand AT&T Labs with a brief tenure at Cisco Systems. He has worked on data and voice networking servicesdevelopment, network architecture, design, and QoS.Dr. Deshpande was a faculty member of computerscience and electrical engineering at the IndianInstitute of Technology, New Delhi, India, from 1973 to 1982. After spending one year as a visiting facultymember at Pennsylvania State University, he joined BellLabs in 1983.

TEWFIK L. DOUMI is a principal in the Network and Performance Reliability Department atAlcatel-Lucent Bell Labs in Murray Hill,New Jersey. He holds a license in physicsfrom the University of Algiers, Algeria; anM.S. degree in electrical engineering from

Stevens Institute of Technology, Hoboken, NewJersey; and a Ph.D. degree in electrical engineeringfrom the University of Bradford in England. Dr. Doumi’s professional interests are in spectrummanagement and radio engineering techniques fornext-generation wireless systems. He is a member ofthe Alcatel-Lucent Technical Academy and a seniormember of the IEEE.

MARK MADDEN is the regional vice president for Energy Markets in Alcatel-Lucent’sAmericas Region. He is responsible forAlcatel-Lucent’s North American marketstrategy, strategic partnerships, andbusiness development in the utility, oil,

and gas markets. Mr. Madden joined Alcatel-Lucent in1996. He has over 25 years experience with leadingcompanies in the information and communicationstechnologies industry and has been actively engagedproviding consulting to various customers within the electric utility sector on mission-criticaltelecommunications technologies for the last five years.


TIM MEW is a member of Alcatel-Lucent’s Global Services team, responsible for defining,developing, and managing complexsystems integration services solutions forthe energy and utilities sector. Prior to this,he was the head of the Solution Design and

Innovation team in Australasia, focusing on railways,highways, oil and gas, and security services solutions.Mr. Mew’s generalist background has includedarchitecture, technology planning, and servicedevelopment disciplines in the technology areas ofnext-generation networks, VoIP, intelligent networks,PSTN, CTI, Internet and IP, CCTV, e-commerce, andwireless communications. Before joining Alcatel-Lucent, he held diverse roles ranging from architecturemanager, senior engineer, to brand manager and CTOroles in a number of industries including carriers, ISPs,and e-commerce. ◆

Date post:	21-Apr-2018
Category:	Documents
Upload:	dangcong
View:	219 times
Download:	5 times

Communication Network Architecture and Design...

Documents