1 Internet Protocol Based Monitoring and Control System for Iraqi Power Generation Stations Asst. Prof. Nassir N. Khamiss Nahrain University, College of Information Engineering E-mail: [email protected]Asst. Lect. Suhail N. Obaid Foundation of Technical Education, Technical College of Management – Department of Information Technology E-mail: [email protected]Abstract As electrical power industry enters the new era, Computer-based Supervisory Control and Data Acquisition (SCADA) systems have been evolved from standalone, traditional compartmental operations into networked architectures that communicate through computer networks across large distances. The SCADA system that monitors and controls the Iraqi power generation plant suffers from repeated failure in the data transfer via the radio link of the communication network used because of the unavailability and instability of this link which is affected by the military jamming signals and by other external interference signals which degrade the overall system operation and decrease the system stability and performance. Moreover the used serial communication link protocol of typical data rate (300- 19200 bit/second) considered relatively low bit rate protocol especially when planning to add more services to the SCADA system which increase the system limitation. This research addresses the design of an IP based SCADA system and proposes the use of an optical communication backbone as a communication media instead of the radio media because of its very high immunity to external interference signals and high bit rate up to several gigahertz. The optical fiber backbone was used because of its facilities and because it was already installed in Iraq so there is no need to install a new infrastructure which saves the cost of installing a new one. The proposed communication network was simulated and operated successfully. Most of the proposed SCADA system elements were configured, operated and tested experimentally. The proposed services were tested over local and global (over the internet) networks and were operational. Key Words: (SCADA, Automation, Internet Protocol, Control, Data Acquisition). 1. INTRODUCTION Electricity supply is one of the most important utilities that underpin the survival of a nation’s critical infrastructures and services such as natural gas supply, water supply services,
Transcript
1
Internet Protocol Based Monitoring and Control System for Iraqi Power Generation Stations
Asst. Prof. Nassir N. Khamiss
Nahrain University, College of Information Engineering E-mail: [email protected]
Asst. Lect. Suhail N. Obaid
Foundation of Technical Education, Technical College of Management –
As electrical power industry enters the new era, Computer-based Supervisory Control and
Data Acquisition (SCADA) systems have been evolved from standalone, traditional
compartmental operations into networked architectures that communicate through computer
networks across large distances. The SCADA system that monitors and controls the Iraqi
power generation plant suffers from repeated failure in the data transfer via the radio link of
the communication network used because of the unavailability and instability of this link
which is affected by the military jamming signals and by other external interference signals
which degrade the overall system operation and decrease the system stability and
performance. Moreover the used serial communication link protocol of typical data rate (300-
19200 bit/second) considered relatively low bit rate protocol especially when planning to add
more services to the SCADA system which increase the system limitation. This research
addresses the design of an IP based SCADA system and proposes the use of an optical
communication backbone as a communication media instead of the radio media because of its
very high immunity to external interference signals and high bit rate up to several gigahertz.
The optical fiber backbone was used because of its facilities and because it was already
installed in Iraq so there is no need to install a new infrastructure which saves the cost of
installing a new one. The proposed communication network was simulated and operated
successfully. Most of the proposed SCADA system elements were configured, operated and
tested experimentally. The proposed services were tested over local and global (over the
internet) networks and were operational.
Key Words: (SCADA, Automation, Internet Protocol, Control, Data Acquisition).
1. INTRODUCTION Electricity supply is one of the most important utilities that underpin the survival of a
nation’s critical infrastructures and services such as natural gas supply, water supply services,
2
transportation, telecommunications, financial services and healthcare services [1]. SCADA
stands for (Supervisory Control And Data Acquisition). It generally refers to an industrial
control system which includes manufacturing, production, power generation, fabrication, and
refining [2]. SCADA systems have evolved over the past decades, from standalone,
compartmental operations into networked architectures that communicate across large
distances. In addition, their implementations have migrated from custom hardware and
software to standard hardware and software platforms. These changes have led to reduced
development, operational, and maintenance costs as well as providing executive management
with real time information that can be used to support planning, supervision, and decision
making. These benefits are made possible by the use of standard hardware and software in
SCADA systems combined with improved communication protocols and increased
connectivity to outside networks, including the Internet [3].
Automation of power systems has increasingly been adopted by power utilities worldwide in
recent years as part of the efforts to provide a more reliable supply to its customers and to
enhance operational efficiency [4].
Figure (1) shows the operation of a simple computer based industrial control system for plant
monitoring and control by the system operator [5].
Figure (1) Computer based industrial control system operation [5].
SCADA systems encompass the transfer of data between a SCADA central host computer,
referred to as Master Terminal Unit (MTU), and a number of Remote Terminal Units (RTUs)
located at separate distances from the MTU [3]. The RTU is typically a layer of equipment
between the remote sensors and instruments and the central computer. This intermediate
equipment exists on the remote side and is connected to the sensors and field instruments.
1.1 SCADA System Generations There are three generations of SCADA systems (Monolithic, Distributed and Networked
generation) [6]; which have been explored in the following sub sections:
Signals from sensors
Information Commands
Signals to actuators
Operator
Computer
Plant
3
A. First Generation: Monolithic
SCADA systems were standalone systems with virtually no connectivity to other systems
as shown in figure (2). The wide-area networks (WANs) that were implemented to
communicate with remote terminal units (RTUs) were designed with a single purpose in
mind: that of communicating with RTUs in the field and nothing else. In addition, the WAN
protocols in use today were largely unknown at the time. The protocols in use on SCADA
networks were developed by the vendors of RTU equipment and were often treated as
proprietary. This meant that, for some RTU protocols, no other vendors were allowed to build
equipment that communicated via these protocols.
Figure (2) First generation SCADA architecture.
B. Second Generation: Distributed
The next generation of SCADA systems began to take advantage of developments and
improvement in system miniaturization and local-area networking (LAN) technology to
distribute the processing across multiple systems. Multiple stations, each with a specific
function, were connected to a LAN and shared information with each other in real-time.
These stations were typically of the minicomputer class, rather than mainframes, and were
smaller and less expensive than their first generation predecessors. This generation of
SCADA systems depicted in figure (3).
Figure (3) Second generation SCADA architecture.
4
C. Third Generation: Networked The third generation of SCADA master station architecture is closely related to that of the second generation, with the primary difference being that of open system architecture rather than a vendor-controlled, proprietary environment. There are still multiple networked systems sharing master station functions. There are still RTUs utilizing protocols that are vendor proprietary. The major improvement in the third generation is that of opening the system architecture, utilizing open standards and protocols and making it possible to distribute SCADA functionality across a WAN and not just a LAN as shown in figure (4). As they have moved to “open” or “off-the-shelf” systems, SCADA vendors have gradually go out of the hardware development business. This allows SCADA vendors to concentrate their development in an area where they can add specific value to the system, that of SCADA master station software. Open systems do greatly improve the connectivity to external systems, but there can still be interconnection limitations, even with systems or equipment that all adhere to the same standards. The major improvement in third-generation SCADA systems comes from the use of WAN
protocols, such as TCP/IP, for communication between the master station and
communications equipment.
Figure (4) Third generation SCADA architecture.
Systems controlling critical infrastructure for generating, transmitting, distributing, storing,
and utilizing energy as well as for processes in manufacturing are no longer isolated. The
drive towards networked industrial control systems is due to several factors. Integration of
geographically distributed assets through centralized control improves agility in responding to
supply and demand fluctuations, reduces cost of operations and enables process efficiencies
unachievable in the past. [7]
2. Existing Iraqi Power Plant SCADA System The existing Iraqi power plant SCADA system will be described in the following
subsections that will help to get the benefits behind the proposed one.
2.1 Topology The existing Iraqi power plant topology is a point-to-multipoint (Star) topology so that it
is divided into three control centers (North at Kirkuk control center, Middle at Baghdad
5
Control Center and South at Basrah Control Center). The control centers consider as a point
where the master radio modem station exists and the remote power stations modems considers
as points (multi-points) connected to the control center radio master station (point) via radio
links in case of radio media where as in PLC (Programming Logic Controller) the PLC master
modem at the control center plays the role of point and the remote PLC modems of the remote
power station acts as a (multi-points). It is a Master/Slave Client/Server architecture topology.
2.2 Architecture The Iraqi power plant SCADA system is of second generation architecture (i.e. it is
distributed system) the control center servers connected via LAN (Local Area Network)
among each other using IP protocol while the whole control center elements connected to the
remote power stations via an industrial automation serial asynchronous tele-control protocol
which is the IEC 60870-5-101.
The control center servers connected with each other via an IP based LAN and with external
terminals via radio link based network and exchanges the data serially with them and the
interface layer between these two networks is a Cisco router which converts the radio link
serial data into an IP packets to the local control center network but it cannot be considered as
IP based SCADA system since the conversion of the protocol is just for connection
compatibility between the two networks elements. Figure (5) depicts the general connection
criteria of the Iraqi power plant radio link based SCADA system.
Figure (5) General connection criteria of the Iraqi power plant radio link based SCADA system.
2.3 Communication Network Media The communication network media between the control center and the remote stations is
either a radio link based communication network in the (VHF/UHF) band using radio
6
modems or using the PLC (Power Line Carrier) technology. Figure (6) shows the general
communication network architecture of the radio link based network.
Figure (6) General architecture of the radio link b ased SCADA system.
2.4 Communication Protocol The communication protocol used in the Iraqi power plant SCADA system is the industrial
standard for power system automation IEC 60870-5-101. This is asynchronous serial tele-
control protocol having the following features:
1- Supports unbalanced (master initiated message) and balanced (master/slave initiated
message) modes of data transfer.
2- Link address and ASDU addresses are provided for classifying the end station and different
sectors under the same.
3- Cyclic and spontaneous data updating schemes are provided.
2.5 Limitations The limitations for the current SCADA system of Iraqi power plant can be addressed as
bellow:
1- It is a Master/Slave architected system (i.e. the automation can only be initiated by a
request from the master station to slave terminals). In this case the master station should be
placed only at the control center in this case the SCADA system is a standalone system with
virtually no connectivity to other system. The fail of the control center means the failure of
the whole system.
WAN connection via VHF/UHF Links
WAN connection via VHF/UHF Links
7
2- The SCADA operators can only do the automation operations from stations connected
directly to the SCADA master station via WAN which limits the remote ability of the
automation process.
3- The communication network used for data exchange is either the radio link in the
(VHF/UHF) bands or using PLC technology and both of them has its own limitations as
bellow:
2.5.1 Radio Link communication network
a. Low channel capacity.
b. Low digital data bit rate.
c. Limited transmission techniques available (does not support Ethernet connectivity).
d. Highly effected (instability) by external interference (Military jamming signals, other non
licensed radio systems working with shared bands, weather changes etc…).
e. Low security level techniques available.
f. Multi-point operation restricts data speed compared to Point-to-Point UHF or dedicated
paths between stations.
2.5.2 PLC communication network
a. Not independent of the power distribution system.
b. Carrier frequencies often not protected on a primary basis.
c. Inherently few channels available.
d. Expensive on a per channel basis compared to microwave (normally, over four channels).
e. Will not propagate through open disconnects.
4- Limited expandability (vendor proprietary) compared to other open systems which is not
dependent on specific vendor.
5- The addressing scheme depends on the link address of the end station. This addressing
scheme decrease the system connectivity to other infrastructures (LAN/WAN) and the global
internet.
6- The GUI (Graphical User Interface) of this system is not WEB based software which
limits the operability of the system from a remote station and decreases the operation
flexibility and connectivity to other systems.
3. PROPOSED IRAQI POWER PLANT SCADA SYSTEM. The proposed SCADA system for Iraqi power plant have the same topology for the
existing one (i.e. point-to-multipoint) star connection between the control center(s) and the
remote power stations. The main developments will be on the system architecture,
communication protocol, automation software, addition of new services and enhances the
security and integrity level of the whole system.
8
3.1 Topology
The proposed system topology is peer-to-peer (Multipoint to Multipoint) from any point to
any point within the network Networked topology (Third generation). The server works as a
peer and the clients (network elements) work as peers and share network resources. This
topology will allow all network elements to remain available even though the server will goes
down, but the service offered by the damaged server will no longer be available. The
proposed solution is to add extra (Backup) servers so that if the first server fails the backup
one will provide the missing service (i.e. each control center have backup server responsible
for the area it covers) in active standby mode with automatic changing over technique.
Another strategy has been proposed that is to route the status traffic to another server in other
regions which will grantees the monitoring of the network elements and providing the
required server's service. In this case the backup server will holds up the responsibility of the
region it covers and the region which has been lost its server (covers two regions).
3.2 Architecture The proposed system architecture is the third generation architecture criteria (Networked
architecture). The proposal is to make all the SCADA network an open computer system with
no dependency on any industrial or vendor proprietary systems. The proposal is to distribute
the automation load among rather than one control center for load reduction and increasing
redundancy assurance. Three control centers proposed one for middle region, one for north
region and one for south region. General representation for the proposed system architecture
for the three regions is shown in figure (7), where the red line shows the redundant path
between the north and south control centers. While the physical implementation of the
proposed architecture is shown in figure (8) using the national fiber optic network.
3.3 Communication Network Media The most affected part of changing to third generation architecture is the communication
network since it will work on different protocol which enforces to change a lot of system
communication technologies and parameters setting. As known the Iraqi infrastructure uses
the optical network as its backbone for information exchange, which are available at the main
nodes of communications centers, switching system, that are not far from the electrical power
stations. Such an infrastructure can be used for power plant data exchange. In this work there
are two proposed solutions; the first one using wired communications (Optical fiber or DSL).
The other one by using the wireless communications via Microwave system. Such proposals
can connect the power plant to the nearest communication node to join the SCADA network.
The proposed SCADA system network is shown in figure (7).
9
3.4 Communication Protocol The proposed communication network to implement the third generation network
architecture can be achieved with the use of the Internet Protocol instead of the industrial
asynchronous serial protocol IEC 60870-5-101, which is also n open protocol and using this
protocol will make the whole system an open developable network system.
3.5 VAS services The Value Added Services (VAS) considered as an extra services offered with the SCADA
system default services. These services increase the system functions and operations. These
services will be described in the following subsections.
3.5.1 Video over IP service
This service enables security personnel, fire control centers, police headquarters, terrorism
response centers, and operations staff to use the system from both onsite and remote locations.
This flexibility enables vastly accelerated and highly coordinated responses to fire, sabotage,
terrorism, theft, and normal operational incidents [7].
3.5.2 Quality of Service (QoS) service
It provides statistical analysis for the quality of service by generating statistical reports on
any element within the SCADA network for any required period of time.
The SCADA system must include the reporting of the collected data from the RTUs and make
the required analysis on the recorded data. These KPIs (Key Performance Indications) must
be available at the operator immediately when the occurred at any time for daily, weekly and
yearly basis in summary format. The SCADA software should be capable of reporting this
information. Operation staff must access historical and trend data day to day. The data
analysis area of the SCADA system is another very important area and one sometimes
overlooked to make the system operationally effective [9].
This is a powerful statistical tool that can read the log files with (.CSV) format submitted by
ALCATEL-LUCENT France Company called QoSAC server (Quality of Service Analysis
Center).
This server has high capabilities to store and handle the log data files and performs statistical
operations on these data with a high speed and a very high accuracy over any needed period
of time.
10
Figure (7) General proposed architecture for SCADA system network.
11
Figure (8) Physical implementation diagram of the o ptical fiber based SCADA
Many companies consider the use of the internet for supervisory control and data
acquisition is to provide access to real-time data display, alarming, trending, and reporting
from remote equipment [10].
Sometimes SCADA operators need to do all SCADA operations even they use networks that
are not connected to the SCADA network. The virtual private network (VPN) connections
enable remote operators to operate the SCADA system from locations out of SCADA
network using the internet as a communication media. The VPN link is protected by means
Fiber cable to other regional router
Fiber cable from optical multiplexer to the regional gateway router
Fiber cable to other regional router
Cisco router (Regional Gateway)
Fiber patch panel
Fiber patch panel
Cisco router (Regional Gateway)
Generation station switch
National Fiber Optic
Network
12
of encryption and authentication providing rather reliable and secure data transmission paths
[11].
A VPN client uses special TCP/IP-based protocols, called tunneling protocols, to make a
virtual call to a virtual port on a VPN server. In a typical VPN deployment, a client initiates a
virtual point-to-point connection to a remote access server over the Internet. The remote
access server answers the call, authenticates the caller, and transfers data between the VPN
client and the organization’s (SCADA network) private network. To emulate a point-to-point
link, data is encapsulated, or wrapped, with a header. The header provides routing information
that enables the data to traverse the shared or public network to reach its endpoint. To emulate
a private link, the data being sent is encrypted for confidentiality. Packets that are intercepted
on the shared or public network are indecipherable without the encryption keys. The link in
which the private data is encapsulated and encrypted is known as a VPN connection. The
connection topology of the central control center NEs (Network Elements) is shown in figure
(9).
3.6 Features
The new proposed communication network will provide the following:
1- High bandwidth since it proposes the use of the standard high band width communication
technologies (Microwave, Optical Fiber Network).
2- Standard data encapsulation using the standard data communication protocol (Internet
Protocol IP) among all network elements.
3- More secure since it uses the standard modern technologies of computer network
technologies (i.e. fiber optic cables) and high reliable security algorithms (CHAP Challenge
Hand shake Application Protocol, PPTP Point to Point Tunneling Protocol etc..).
4- Distributed control topology so that the control and automation will be distributed among
the proposed control centers/servers and not just central control center/ standalone server
handling all the data to increase the system reliability.
5- Increases the system connectivity since the use of the IP protocol increases the system
connectivity to other infrastructures such as the internet/intranet.
6- Increase the services offered by the system such as VPN (Virtual Private Network) service,
QoSAC (Quality of Service Analysis Center service, VoIP (Voice over IP) service, Video
conferencing/Monitoring using IP cameras, Web based software service.
3.7 System Simulation
The proposed SCADA network simulation diagram is shown in figure (10). This simulation
has been built using Boson NetSim for CCNP (Cisco Certified Network Professional)
software. The proposed web site map has been depicted in figure (11) for the automation of
the power generation stations.
13
4. Results The practical results obtained from the experimental work and simulation work has been
explained in the following subsections.
4.1 Web based SCADA Software This software can be provided by the WEB server can be considered as regional control
center server's software. While the monitoring and control page of the generation station
depicted in figure (12).
4.2 The VPN server test results After configuring the Cisco router (2621) as a VPN sever and connecting it to a class B
private IP network with network ID (192.168.0.0) and connecting the other side of it to the
internet and assigning a public IP address to the interface connecting to the internet, a remote
test is accomplished by using the internet as a communication media and configuring the
remote host, then establish a tunnel connection from the remote site to the VPN server. The
connection has been achieved successfully and login to the private network remotely. The
remote client IP address (213.188.74.118), the VPN server was fully functional for the planed
operation. Figure (13) shows the login window (VPN connection authentication window) and
the registration confirmation message after successfully logging to the VPN server.
The (ipconfig) implementation in the VPN client is to display the client's IP address after
logging into the private network via the VPN server to be (192.168.52.205) and a ping
command to the QoSAC server which connects to this network with an IP address
(172.16.18.19) has been implemented successfully as shown in figure (14).
14
Figure (9) Connection of the central control center network elements.
15
Figure (10) Simulation diagram for the proposed IP based SCADA network.
16
Figure (11) Website map of the proposed IP based SC ADA system
Figure (12-a) RTU560 monitoring and control pages ( Monitoring page)
IP Camera home page
RTU home page (Monitoring and Control page)
RTU home page (Monitoring and Control page)
RTU home page (Monitoring and Control page)
Generation station home page (Central)
Generation station home page (South)
Generation station home page (North)
Regional HMI server home page (Central)
Regional HMI server home page
(South)
Regional HMI server home page
(North)
Home page (Login authentication page)
IP Camera home page
IP Camera home page
17
Figure (12-b) RTU560 monitoring and control pages ( Control page).
(a) (b)
Figure (13) Connecting to remote SCADA network via the VPN server (a) login
window (b) registration confirmation message to the private IP network.
18
Figure (14) Displaying the VPN client's IP address before and after the connection to the remote SCADA network.
5. Conclusions During the implementation of the system, a number of conclusions have been considered
based on the practical and simulation results obtained from the implemented system and the
most important ones are listed below:
1. The implemented system was cost effective solution compared with other approaches
to build such a system. The central HMI server needs relatively very low resources to
achieve its task.
2. The use of RTU560 system is a very compatible and reliable solution since the RTU
accepts the interfacing with the SCADA network by either the Ethernet or serial data
interface module which increase the system compatibility/connectivity. The RTU560
system works as a WEB server so it can be accessed using any internet browser.
3. The use of the internet protocol (IP) enables the interfacing of the system to the
internet/intranet and increase the system connectivity with any IP based systems.
4. The particular characteristics of the fiber optic network such as low attenuation, high
bandwidth, small physical cross section, electromagnetic interface immunity, and
security, make it the most suitable transmission medium for the power generation
stations automation, control, protection and monitoring functions. Moreover already
19
installed optical fiber infrastructure in Iraq saves the cost behind installing other
communication media. Further it supports new services and functions that drive
increased bandwidth and time latency requirements.
5. The value added services (VAS) proposed to be added to the default system services
increases the system operability and performance analysis.
6. The system (MTU, RTU, HMI and the communication network) is easy to use and
setup. The knowledge base needed by the system administrators and operators is very
common in the IT field. There are many large companies that provide courses and
certifications which cover most of knowledge required to setup and use the
implemented system.
7. The implemented system proved to be able to work continuously for a very long time
without breakdown.
6. References [1] Saifur Rahman, Manisa Pipattanasomporn and Yonael Teklu, " Intelligent Distributed
Autonomous Power Systems (IDAPS) ", IEEE PES Summer Meeting in Tampa, Florida. June
24-28, 2007.
[2] Kostas Kalaitzakis, “Development of a data acquisition system for remote monitoring of renewable energy systems”, Elsevier Ltd., 2003. [3] Ronald L. Krutz, “Securing SCADA Systems”, Wiley Publishing Inc., 2006.
[4] Eng-Kiat Chan and Horst Ebenhoh, “The Implementation and Evolution of a SCADA
System for a Large Distribution Network”, IEEE Transactions on Power Systems, Vol. 7, No.
1, February 1992.
[5] M. A. Laughton and D. J. Warne, “Electrical engineer’s reference book”, Newnes, 2003.
[6] Robert H. Mcclanahan, “SCADA and IP: Is Network Convergence Really Here?”, White
paper, IEEE Industry Applications, 2003.
[7] Juniper Networks Inc., “Architecture For Secure Scada And Distributed Control System Networks”, White Paper, July 2010. [8] Industrial Video Control Inc., “Need An Industrial Video System Solution?”, White paper, IV&C Ltd., 2006. [9] Adrian Neisbet, “Modern SCADA Technologies and Methods of Integration”, White
paper, RAD-TEL Systems Inc., 2005.
[10] Donald Wallace, “How to put SCADA on the Internet”, White paper, M2M Data Corp.,
2003.
[11] Mikael Nordman et. al., “A TCP/IP Based Communication Architecture for Distribution
Network Operation and Control”, White paper, CIRED Inc., 17th International Conference on
