Document Number C07A10/62.04.00.00
Document Group SkyWay
Document Title Administrator Manual
Revision/Date 1.4.0 / 20091014
Applies to HW/SW Software
Applies to O/S Linux
Job Code/Description C07A10 INDIAN AIRPORTS
Customer A.A.I.
File Name /tt/file_convert/552a5e5355034666428b45fc/document.doc
INDIAN AIRPORTS SKYWAY Administrator Manual
INDIAN AIRPORTS:GOA – GAYA
BHUBANESWAR
VIZAG – UDAIPUR
NAGPUR – TRICHY
PATNA – RANCHI
SRINAGAR
TRIVANDRUM
FLIGHT INFORMATION DISPLAY SYSTEM
SkyWay
ADMINISTRATOR MANUAL
Page II
INDIAN AIRPORTS SKYWAY Administrator Manual
Revision: 1.4.0 October 2009 Rif.: C07A10
Page III
INDIAN AIRPORTS SKYWAY Administrator Manual
INDEX
1 INTRODUCTION................................................................................................1
2 CONTROL SYSTEM............................................................................................2
3 FUNCTIONS OF EACH SERVER.........................................................................16
3.1 DATABASES (LB SERVERS).............................................................................................173.1.1 LINUX CLUSTER.......................................................................................................183.1.2 DRBD......................................................................................................................22
3.2 ASA SERVER...................................................................................................................263.2.1 RAID SOFTWARE ON DATABASES DISKS.................................................................283.2.2 WHAT TO DO IF A DATABASE DISK IS BROKEN.......................................................28
3.3 WEB SERVER..................................................................................................................30
4 HOW TO ….....................................................................................................31
4.1 CHANGE CONSOLE …..........................................................................................................314.2 STOP / RESTART A SERVER …...........................................................................................324.3 KNOW IF THE SERVER IS THE MASTER ….................................................................................33
5 SOLARI APPLICATIONS...................................................................................35
5.1 DATABASE SERVER (L.B. SERVER)..................................................................................355.1.1 Main Processes.......................................................................................................35
5.2 ASA SERVER...................................................................................................................38
6 SYSTEM MONITOR..........................................................................................39
7 LOGO INSTALLATION......................................................................................42
8 LOG FILES.....................................................................................................43
9 LINK WITH AFTN.................................................................................................................45
Page IV
INDIAN AIRPORTS SKYWAY Administrator Manual
1 INTRODUCTION
This document is designed for users having skill of system administrator and want to be a
QUICK REFERENCE MANUAL
to be utilized in case of abnormal behaviour of the system.
For this reason it’s organized as a set of sheets for quick and easy consulting, one argument per page.
IMPORTANT NOTE
All the pictures reported in this manual are referred to JAIPUR Airport system, but can be useful for all other airports.
Page 1
INDIAN AIRPORTS SKYWAY Administrator Manual
2 CONTROL SYSTEM
The FIDS system is composed, in addition to peripheral devices, by a central control system, made by one computer below described, installed in one rack:
This is the minimum configuration, a single PC is used to run Database Server and Application Server software; an additional PC (workstation) is used to manage the system and for operational procedures. As the number of displays starts to grow, an additional PC may be dedicated to run the Application Server software, while the first now acts only as a Database Server. Additional Application Servers may be added to the system, in order to handle an even increased number of displays.
Page 2
RACK
switch device
DB1- LB1- AS1 (all in 1 machine)
DB1- LB1- AS1 (stand-by)
WEB
ASA
ITA (IVRS SERVER)
INDIAN AIRPORTS SKYWAY Administrator Manual
PATNA AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1pat 172.26.73.13255.255.0.0
192.168.73.13255.255.255.0
172.26.73.1 (*)255.255.0.0
172.26.73.4 (*)255.255.0.0
172.26.73.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2pat 172.26.73.14255.255.0.0
192.168.73.14255.255.255.0
172.26.73.1 (*)255.255.0.0
172.26.73.4 (*)255.255.0.0
172.26.73.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asapat 172.26.73.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webpat 172.26.73.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itapat 172.26.73.7255.255.0.0
IVRS Server Windows
Page 3
INDIAN AIRPORTS SKYWAY Administrator Manual
RANCHI AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1ixr 172.26.74.13255.255.0.0
192.168.74.13255.255.255.0
172.26.74.1 (*)255.255.0.0
172.26.74.4 (*)255.255.0.0
172.26.74.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2ixr 172.26.74.14255.255.0.0
192.168.74.14255.255.255.0
172.26.74.1 (*)255.255.0.0
172.26.74.4 (*)255.255.0.0
172.26.74.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asaixr 172.26.74.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webixr 172.26.74.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itaixr 172.26.74.7255.255.0.0
IVRS Server Windows
Page 4
INDIAN AIRPORTS SKYWAY Administrator Manual
SRINAGAR AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1sxr 172.26.75.13255.255.0.0
192.168.75.13255.255.255.0
172.26.75.1 (*)255.255.0.0
172.26.75.4 (*)255.255.0.0
172.26.75.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2sxr 172.26.75.14255.255.0.0
192.168.75.14255.255.255.0
172.26.75.1 (*)255.255.0.0
172.26.75.4 (*)255.255.0.0
172.26.75.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asasxr 172.26.75.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
websxr 172.26.75.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itasxr 172.26.75.7255.255.0.0
IVRS Server Windows
Page 5
INDIAN AIRPORTS SKYWAY Administrator Manual
TRIVANDRUM AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1trv 172.26.82.13255.255.0.0
192.168.82.13255.255.255.0
172.26.82.1 (*)255.255.0.0
172.26.82.4 (*)255.255.0.0
172.26.82.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2trv 172.26.82.14255.255.0.0
192.168.82.14255.255.255.0
172.26.82.1 (*)255.255.0.0
172.26.82.4 (*)255.255.0.0
172.26.82.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asatrv 172.26.82.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webtrv 172.26.82.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itatrv 172.26.82.7255.255.0.0
IVRS Server Windows
Page 6
INDIAN AIRPORTS SKYWAY Administrator Manual
GAYA AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1gay 172.26.72.13255.255.0.0
192.168.72.13255.255.255.0
172.26.72.1 (*)255.255.0.0
172.26.72.4 (*)255.255.0.0
172.26.72.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2gay 172.26.72.14255.255.0.0
192.168.72.14255.255.255.0
172.26.72.1 (*)255.255.0.0
172.26.72.4 (*)255.255.0.0
172.26.72.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asagay 172.26.72.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webgay 172.26.72.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itagay 172.26.72.7255.255.0.0
IVRS Server Windows
Page 7
INDIAN AIRPORTS SKYWAY Administrator Manual
GOA AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1goi 172.26.71.13255.255.0.0
192.168.71.13255.255.255.0
172.26.71.1 (*)255.255.0.0
172.26.71.4 (*)255.255.0.0
172.26.71.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2goi 172.26.71.14255.255.0.0
192.168.71.14255.255.255.0
172.26.71.1 (*)255.255.0.0
172.26.71.4 (*)255.255.0.0
172.26.71.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asagoi 172.26.71.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webgoi 172.26.71.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itagoi 172.26.71.7255.255.0.0
IVRS Server Windows
Page 8
INDIAN AIRPORTS SKYWAY Administrator Manual
BHUBANESWAR AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1bbi 172.26.70.13255.255.0.0
192.168.70.13255.255.255.0
172.26.70.1 (*)255.255.0.0
172.26.70.4 (*)255.255.0.0
172.26.70.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2bbi 172.26.70.14255.255.0.0
192.168.70.14255.255.255.0
172.26.70.1 (*)255.255.0.0
172.26.70.4 (*)255.255.0.0
172.26.70.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asabbi 172.26.70.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webbbi 172.26.70.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itabbi 172.26.70.7255.255.0.0
IVRS Server Windows
Page 9
INDIAN AIRPORTS SKYWAY Administrator Manual
VIZAG AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1vtz 172.26.69.13255.255.0.0
192.168.69.13255.255.255.0
172.26.69.1 (*)255.255.0.0
172.26.69.4 (*)255.255.0.0
172.26.69.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2vtz 172.26.69.14255.255.0.0
192.168.69.14255.255.255.0
172.26.69.1 (*)255.255.0.0
172.26.69.4 (*)255.255.0.0
172.26.69.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asavtz 172.26.69.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webvtz 172.26.69.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itavtz 172.26.69.7255.255.0.0
IVRS Server Windows
Page 10
INDIAN AIRPORTS SKYWAY Administrator Manual
UDAIPUR AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1udr 172.26.68.13255.255.0.0
192.168.68.13255.255.255.0
172.26.68.1 (*)255.255.0.0
172.26.68.4 (*)255.255.0.0
172.26.68.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2udr 172.26.68.14255.255.0.0
192.168.68.14255.255.255.0
172.26.68.1 (*)255.255.0.0
172.26.68.4 (*)255.255.0.0
172.26.68.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asaudr 172.26.68.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webudr 172.26.68.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itaudr 172.26.68.7255.255.0.0
IVRS Server Windows
Page 11
INDIAN AIRPORTS SKYWAY Administrator Manual
NAGPUR AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1nag 172.26.67.13255.255.0.0
192.168.67.13255.255.255.0
172.26.67.1 (*)255.255.0.0
172.26.67.4 (*)255.255.0.0
172.26.67.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2nag 172.26.67.14255.255.0.0
192.168.67.14255.255.255.0
172.26.67.1 (*)255.255.0.0
172.26.67.4 (*)255.255.0.0
172.26.67.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asanag 172.26.67.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webnag 172.26.67.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itanag 172.26.67.7255.255.0.0
IVRS Server Windows
Page 12
INDIAN AIRPORTS SKYWAY Administrator Manual
TRICHY AIRPORT
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1trz 172.26.66.13255.255.0.0
192.168.66.13255.255.255.0
172.26.66.1 (*)255.255.0.0
172.26.66.4 (*)255.255.0.0
172.26.66.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2trz 172.26.66.14255.255.0.0
192.168.66.14255.255.255.0
172.26.66.1 (*)255.255.0.0
172.26.66.4 (*)255.255.0.0
172.26.66.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asatrz 172.26.66.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webtrz 172.26.66.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itatrz 172.26.66.7255.255.0.0
IVRS Server Windows
Page 13
INDIAN AIRPORTS SKYWAY Administrator Manual
JAIPUR AIRPORT (DEMO ONLY)
Hostname “physical” IP address
“virtual” IP address
Description Operating System
lb1jai 172.26.111.13255.255.0.0
192.168.111.13255.255.255.0
172.26.111.1 (*)255.255.0.0
172.26.111.4 (*)255.255.0.0
172.26.111.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #1
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
lb2jai 172.26111.14255.255.0.0
192.168.111.14255.255.255.0
172.26.111.1 (*)255.255.0.0
172.26.111.4 (*)255.255.0.0
172.26.111.21 (*)255.255.0.0
(*) Only if MASTER
LoadBalancer #2
(DB Server)(AREA Server)
Linux(RedHat Oracle
Enterprise Edition)
asajai 172.26.111.5255.255.0.0
ASA Server Linux(RedHat Oracle
Enterprise Edition)
webjai 172.26.111.6255.255.0.0
WEB Server Linux(RedHat Oracle
Enterprise Edition)
itajai 172.26.111.7255.255.0.0
IVRS Server Windows
In the next page the hosts file of JAIPUR Airport is represented.
Page 14
INDIAN AIRPORTS SKYWAY Administrator Manual
# Do not remove the following line, or various programs# that require network functionality will fail.127.0.0.1 localhost.localdomain localhost########################################################### JAIPUR######################################################### ALIAS SERVERS172.26.111.1 lbsrv lbsrv.solari master.solari MASTERALIAS172.26.111.4 LOADBALANCER ntpsrv172.26.111.5 asajai asajai.solari asa asaserver172.26.111.6 webjai webjai.solari172.26.111.7 itajai itajai.solari172.26.111.21 MASTERAS as1jai## LB SERVERS172.26.111.13 lb1jai lb1jai.solari172.26.111.14 lb2jai lb2jai.solari192.168.111.13 lb1jaipriv lb1jaipriv.solari192.168.111.14 lb2jaipriv lb2jaipriv.solari## FIDS WORKSTATIONS172.26.111.51 wks1 wks1.solari172.26.111.52 wks2 wks2.solari172.26.111.53 wks3 wks3.solari172.26.111.54 wks4 wks4.solari172.26.111.55 wks5 wks5.solari
Page 15
INDIAN AIRPORTS SKYWAY Administrator Manual
3 FUNCTIONS OF EACH SERVER
The 2 databases are IBM XSeries-346 with 3HD (Software RAID 5).
They run:
The database software (ORACLE Application ver. 10.1.0.3) (working as Data Base Server)
Application programs (working as Load-balancer)
Application programs (working as Area Server)
The two LBs work in stand-by: at any time only one of them is active (MASTER) while the other (SLAVE) is ready to take the control in case of master failure.
The MASTER, in addition to its own IP address gets 2 IP resources (HA resources).
The MASTER is always seen from the outside as 172.26.111.1 (lbsrv) and/or 172.26.111.4 (LOADBALANCER), doesn’t matter if master is LB1 or LB2.
Page 16
INDIAN AIRPORTS SKYWAY Administrator Manual
3.1DATABASES (LB SERVERS)
The switch between MASTER and SLAVE is provided by Linux Cluster Suite RH4 – UPD4 software.
The MASTER carries out also other functions:
NFS server (peripheral devices get logos from the master through nfs).
SAMBA server (for viewing, modifying logo from a Windows workstation).
DNS server (for peripheral devices only).
WEB server “apache 2.x”. The apache server, configured in “pre-fork mode” is used for providing “devices subscription messages exchange”.
ANY MISCONFIGURATON OF APACHE SERVICE WILL AFFECT THE SYSTEM STABILITY.
Page 17
INDIAN AIRPORTS SKYWAY Administrator Manual
3.1.1 LINUX CLUSTER
3.1.1.1 INTRODUCTION
Red Hat Cluster Manager allows administrators to connect separate systems (called members or nodes) together to create failover clusters that ensure application availability and data integrity under several failure conditions. Administrators can use Red Hat Cluster Manager with database applications, file sharing services, web servers, and more.
To set up a failover cluster, you must connect the member systems (often referred to simply as members or nodes) to the cluster hardware, and configure the members into the cluster environment. The foundation of a cluster is an advanced host membership algorithm. This algorithm ensures that the cluster maintains complete data integrity at all times by using the following methods of inter-member communication:
Network connections between the cluster systems for heartbeat
Shared state on shared disk storage to hold cluster status
To make an application and data highly available in a cluster, you must configure a service (such as an application and shared disk storage) as a discrete, named group of properties and resources to which you can assign an IP address to provide transparent client access. For example, you can set up a service that provides clients with access to highly-available database application data.
You can associate a service with a failover domain, a subset of cluster members that are eligible to run the service. In general, any eligible member can run the service and access the service data on shared disk storage. However, each service can run on only one cluster member at a time, in order to maintain data integrity. You can specify whether or not the members in a failover domain are ordered by preference. You can also specify whether or not a service is restricted to run only on members of its associated failover domain. (When associated with an unrestricted failover domain, a service can be started on any cluster member in the event no member of the failover domain is available.)
You can set up an active-active configuration in which the members run different services, or a hot-standby configuration in which a primary member runs all the services, and a backup cluster system takes over only if the primary system fails.
Figure 1-1 shows an example of a cluster in an active-active configuration.
Page 18
INDIAN AIRPORTS SKYWAY Administrator Manual
Figure 1-1. Example Cluster in Active-Active Configuration
If a hardware or software failure occurs, the cluster automatically restarts the failed member's services on the functional member. This service failover capability ensures that no data is lost, and there is little disruption to users. When the failed member recovers, the cluster can re-balance the services across the members.
In addition, you can cleanly stop the services running on a cluster system and then restart them on another system. This service relocation capability allows you to maintain application and data availability when a cluster member requires maintenance.
Page 19
INDIAN AIRPORTS SKYWAY Administrator Manual
3.1.1.2 CONFIGURATION
The Linux Cluster configuration file is located under /etc/cluster directory and its name is cluster.conf.
<?xml version="1.0" ?>
<cluster config_version="4" name="lb_jai_cluster">
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="lb1jaipriv" votes="1">
<fence/>
</clusternode>
<clusternode name="lb2jaipriv" votes="1">
<fence/>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices/>
<rm>
<failoverdomains/>
<resources>
<script file="/solids/script/skyway.sh" name="skyway"/>
</resources>
<service autostart="1" name="skyway">
<script ref="skyway"/>
</service>
</rm>
</cluster>
Example of cluster.conf file (JAIPUR).
Page 20
INDIAN AIRPORTS SKYWAY Administrator Manual
The command:
# clustat
gives all information related to Linux Cluster functionality.
[root@lb1jai ~]# clustat
Member Status: Quorate
Member Name Status
------ ---- ------
lb1jaipriv Online, Local, rgmanager
lb2jaipriv Offline
Service Name Owner (Last) State
------- ---- ----- ------ -----
skyway lb1jaipriv started
In the above mentioned example the MASTER is lb1jaipriv machine: the service skyway is started on lb1jaipriv.
The other server, lb2jaipriv, is offline (see hosts file, page 15).
Linux Cluster is therefore working with the 192.168 subnet.
Page 21
INDIAN AIRPORTS SKYWAY Administrator Manual
3.1.2 DRBD
3.1.2.1 INTRODUCTION
The Distributed Replicated Block Device (DRBD) is a software-based, shared-nothing, replicated storage solution mirroring the content of block devices (hard disks, partitions, logical volumes etc.) between servers.
DRBD mirrors data :
In real time. Replication occurs continuously, while applications modify the data on the device.
Transparently. The applications that store their data on the mirrored device are oblivious of the fact that the data is in fact stored on several computers.
Synchronously or asynchronously. With synchronous mirroring, a writing application is notified of write completion only after the write has been carried out on both computer systems. Asynchronous mirroring means the writing application is notified of write completion when the write has completed locally, but before the write has propagated to the peer system.
DRBD's core functionality is implemented by way of a Linux kernel module. Specifically, DRBD constitutes a driver for a virtual block device, so DRBD is situated “right near the bottom” of a system's I/O stack. Because of this, DRBD is extremely flexible and versatile, which makes it a replication solution suitable for adding high availability to just about any application.
Important
DRBD is, by definition and as mandated by the Linux kernel architecture, agnostic of the layers above it. Thus, it is impossible for DRBD to miraculously add features to upper layers that these do not possess. For example, DRBD cannot auto-detect file system corruption or add active-active clustering capability to file systems like ext3 or XFS.
Page 22
INDIAN AIRPORTS SKYWAY Administrator Manual
Figure 1.1. DRBD's position within the Linux I/O stack
3.1.2.2 HOW DOES IT WORK ?
Each device (DRBD provides more than one of these devices) has a state, which can be 'primary' or 'secondary'. On the node with the primary device the application is supposed to run and to access the device (/dev/drbdX; used to be /dev/nbX). Every write is sent to the local 'lower level block device' and to the node with the device in 'secondary' state. The secondary device simply writes the data to its lower level block device. Reads are always carried out locally.
If the primary node fails, heartbeat is switching the secondary device into primary state and starts the application there. (If you are using it with a non-journaling FS this involves running fsck)
If the failed node comes up again, it is a new secondary node and has to synchronise its content to the primary. This, of course, will happen whithout interruption of service in the background.
And, of course, we only will resynchronize those parts of the device that actually have been changed. DRBD has always done intelligent resynchronization when possible. Starting with the DBRD-0.7 series, you can define an "active set" of a certain size. This makes it possible to have a total resync time of 1--3 min, regardless of device size (currently up to 4TB), even after a hard crash of an active node.
Page 23
INDIAN AIRPORTS SKYWAY Administrator Manual
3.1.2.3 UTILITY COMMANDS
The location of Oracle datafiles is /oradata partition.
The package drbd (ver. 0.7.15) takes care that /oradata partition is updated from MASTER to SLAVE machine.
DRBD1 is a block device which is designed to build high availability clusters. This is done by mirroring a whole block device via (a dedicated) network.
What is the scope of drbd, what else do I need to build a HA cluster?
DRBD takes over the data, writes it to the local disk and sends it to the other host. On the other host, it takes it to the disk there.
To check the DRBD status on the 2 DBs the following command can be typed (for example on lb1jai):
# cat /proc/drbd
CASE 1: lb1jai is SLAVE, lb2jai is MASTER
CASE 2: lb1jai is MASTER, lb2jai is SLAVE
1 See www.drbd.org.
Page 24
version: 0.7.15 (api:77/proto:74)
SVN Revision: 1725 build by root@lb2gay, 2005-02-15 12:32:18
0: cs:Connected st:Secondary/Primary ld:Consistent
ns:7348 nr:6094824 dw:6102172 dr:66185 al:0 bm:276 lo:0 pe:0 ua:0 ap:0
version: 0.7.15 (api:77/proto:74)
SVN Revision: 1725 build by root@lb2gay, 2005-02-15 12:32:18
0: cs:Connected st:Primary/Secondary ld:Consistent
ns:7348 nr:6094824 dw:6102172 dr:66185 al:0 bm:276 lo:0 pe:0 ua:0 ap:0
INDIAN AIRPORTS SKYWAY Administrator Manual
CASE 3: lb1jai is MASTER, lb2jai is OFF-LINE
The meaning of the status report is described into the following table.
3.1.2.4 CS Connection status
Represents the connection status between the two hosts; can be:
Connected: hosts are connected;
WFConnection: local host is waiting for connection from remote one;
StandAlone: hosts are in stand alone mode, can’t be established which one has good and correct data.
ST Replication status
The meaning of the phrase is Localhost-status/remotehost-status, and the values can be:
Primary: the server synchronizes the other one;
Secondary: the service is synchronized from the other one;
Unknown: the is no connection with the other one hosts (CS will be Wfconnection);
LD Device status
Represents the status of the shared device:
Consistent: data is good
Unconsisstent: data is not good (this state is very unusual)
Page 25
version: 0.7.15 (api:77/proto:74)
SVN Revision: 1725 build by root@lb2gay, 2005-02-15 12:32:18
0: cs:WFConnection st:Primary/Unknown ld:Consistent
ns:7348 nr:6094824 dw:6102172 dr:66185 al:0 bm:276 lo:0 pe:0 ua:0 ap:0
INDIAN AIRPORTS SKYWAY Administrator Manual
3.2ASA SERVER
ASA server provides the link with the Public Announcements’ Server (PAS). It acts as an Area Server for the “specific” device PAS. As specify before this functionality is Integrated in one single machine that runs Database, Load-balancer, automatic announcements and area server. Below is illustrated the tasks related of:
All the digitalisations are stored in the /asafiles folder that can be reached through SAMBA.
# ls /asafiles
E H (*)
(*) In some airports the following local languages are available:
L = TAMIL (Trichy)
M = MARATHI (Nagpur)
O = ORYA (Bhubaneswar)
T = TELGU (Vizag)
Each sub-folder contains the Language-related files: E=English, H=Hindi …
# ls /asafiles/E
AIR ALP BAS CAU CTY GAD GEN NUN ORA
For each language:
FOLDER Description
AIR AIRLINES files: 9W.wav, IT.wav …
ALP Alphanumerical digits (0 – 9 / A – Z)
BAS Fixed phrases (FLIGHT, IS LANDED …)
CAU DELAY causes
CTY CITIES files : DEL.wav, IXZ.wav, …
GAD GATES
GEN General Announcements Phrases
NUN Numbers
ORA Hours
Page 26
INDIAN AIRPORTS SKYWAY Administrator Manual
You can add a new CTY and /or a new AIRLINE simply adding the related “wav” file in the right folder: i.e. if you want to add Guarulhos Airport (GRU) to the English Language, add a file GRU.wav in the /asafiles/E/CTY folder.
REMARK: the system will logs all “MISSING SEGMENTS” in the file /solwrk/asamissing/YYYYMMDD.
Page 27
INDIAN AIRPORTS SKYWAY Administrator Manual
3.2.1 RAID SOFTWARE ON DATABASES DISKS
The ASA Server has 2 disks each, configured in a RAID-1 mirroring. The OS command to verify what is the situation of the disks is mdadm . For example you may type
# mdadm –detail /dev/md2 /dev/md2: Version : 00.90.01 Creation Time : Mon Jan 23 16:14:17 2006 Raid Level : raid1 Array Size : 29398848 (28.04 GiB 30.10 GB) Device Size : 29398848 (28.04 GiB 30.10 GB) Raid Devices : 2 Total Devices : 2Preferred Minor : 2 Persistence : Superblock is persistent Update Time : Mon Feb 27 16:21:12 2006 State : clean Active Devices : 2Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 UUID : 0ca81f54:95c001ca:5d29d873:d1fd82e5 Events : 0.474345
The disk is made of 3 partions (each disk MUST be partitioned in the same exact way)
/dev/md0 is a swap partition
/dev/md1 is a partion containing data (not currently used)
/dev/md2 is the partion containing all software (basic, Oracle, Application)
Each mdX device is a logical device and is actually composed by two physical devices, i.e. /dev/md2 is made up by /dev/sda1 and /dev/sdb1 (this setup has been performed at the initial OS installation).
3.2.2 WHAT TO DO IF A DATABASE DISK IS BROKEN
If you need to replace a disk do the following (be sure you have an identical disk):
1. Remove the broken disk and insert the new one. It is strongly suggested to perform this operation when the machine is running as slave. You may perform the disk replacement without powering off the machine because the disks are hot swappable.
Page 28
INDIAN AIRPORTS SKYWAY Administrator Manual
2. Run the fdisk utility for the replaced disk,i.e. if the disk replaced is the first, type fdisk /dev/sda and create the partions identical as the ones in the good disk.
3. Once the 2 disks have the same partitions run the command :
mdadm --manage /dev/mdX –a /dev/sdbY2
where X and Y are 0,1,2 (Take great care to enter the correct values for X and Y otherwise you may destroy the RAID). In our specific case the commands are
# mdadm --manage /dev/md0 –a /dev/sda3
# mdadm --manage /dev/md1 –a /dev/sda2
# mdadm --manage /dev/md2 – a /dev/sda1
If the disk to be replaced is the second one just substitute /dev/sda with /dev/sdb.
For more information on mdadm and RAID SOFTTWARE man mdadm.
2 The –a flag adds the disk to the array. If you get an error on this command, try to run mdadm –detail /dev/md2. If you see on last lines that /dev/sda1 is stille present in the raid in faulty status, it must be removed with mdadm –manage /dev/md2 –r /dev/sda1 and then added with the mdadm –a command.
Page 29
INDIAN AIRPORTS SKYWAY Administrator Manual
3.3WEB SERVER
WEB server provides the link with the Internet provider in order to display, on the web site, the run-time flight data.
Pointing with a browser to the following address (example for JAIPUR):
http://172.26.111.6/web/
or
http://lbsrv/web/
is possible to access the FIDS WEB PAGES (see next picture).
Page 30
INDIAN AIRPORTS SKYWAY Administrator Manual
4 HOW TO …
This section will describe some useful Linux commands necessary for the Servers management.
4.1Change console …
How to know which is the current console:
# tty
How to change the current console:
ALT F1 Display console #1
ALT Fn Display console #n
ALT “RIGHT ARROW” Next console
ALF “LEFT ARROW” Previous console
How to loggin out:
# exit
Page 31
INDIAN AIRPORTS SKYWAY Administrator Manual
4.2 STOP / RESTART a Server …
NEVER Shutdown a running sever cutting the POWER (PRESSING the POWER ON/OFF BUTTON). This could severely damage the filesystem.
For shutting down a server type the commend:
# halt
and wait the “system halted” message to appear. Then and only then you can power off the machine.
For rebooting a server type the command:
# reboot
The servers are configured with GRUB as bootloader. Two entries are provided. The default entry is the normal one. The second one, called “MAINTENANCE MODE” is a kind of safe-mode entry.
Selecting the MAINTENANCE MODE the FIDS application WILL NOT RUN on the Server.
Page 32
INDIAN AIRPORTS SKYWAY Administrator Manual
4.3 Know if the server is the Master …
If you are logged in a L.B. SERVER and want to know if that server is the MASTER:
# pidof funhan
will give you the PID of the main MASTER PROCESS if any.
or
# ifconfig
will give you the complete list of the IPs Addresses of the machine. If MASTER ALIASES are present of course that means the machine is the MASTER.
# ifconfigeth0 Link encap:Ethernet HWaddr 00:0D:60:EB:46:0E inet addr:172.26.111.13 Bcast:172.26.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1531864 errors:0 dropped:0 overruns:0 frame:0 TX packets:1120267 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:689572186 (657.6 Mb) TX bytes:235253461 (224.3 Mb) Base address:0x2500 Memory:fbfe0000-fc000000
eth0:0 Link encap:Ethernet HWaddr 00:0D:60:EB:46:0E inet addr:172.26.111.4 Bcast:172.26.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Base address:0x2500 Memory:fbfe0000-fc000000
eth0:1 Link encap:Ethernet HWaddr 00:0D:60:EB:46:0E inet addr:172.26.111.1 Bcast:172.26.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Base address:0x2500 Memory:fbfe0000-fc000000
eth1 Link encap:Ethernet HWaddr 00:0D:60:EB:46:0F inet addr:192.168.111.13 Bcast:192.168.57.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Base address:0x2540 Memory:fbfc0000-fbfe0000
lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:2691588 errors:0 dropped:0 overruns:0 frame:0 TX packets:2691588 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
Page 33
INDIAN AIRPORTS SKYWAY Administrator Manual
RX bytes:944706991 (900.9 Mb) TX bytes:944706991 (900.9 Mb)
NO ONE of the MASTER ALIASES have to be present in the SLAVE Server.
Page 34
INDIAN AIRPORTS SKYWAY Administrator Manual
5 SOLARI APPLICATIONS
5.1DATABASE SERVER (L.B. SERVER)
5.1.1 Main Processes
TTY TASK Feature
1 zr The Z-Row manager task manages memory representation of display
peripherals.
Each display line is internally described by a Z-ROW which contains all the
informations about the physical format of the line and the logic information
to be displayed on.
This block is the interface between input processes, which update the
operational file, and the output processes, which format and send to the
peripheral controllers the informations to be displayed; it actually performs
logical mapping between operational file and peripheral rows, implementing
the rules about priority of information and displays capacity managing for
different types of displays.
2 ckhand It receives commands from the WEB APPLICATION from CHECK-IN and GATES DESK and actives funhan
3 output The output task performs the following functions:
- At initialization time reads the system configurations tables on the DB.
- Then waits for activation by zr. After activation the following actions are
made (according to parameters passed by zr):
The record, associated to the z-row, is read from DB and then
translated into the specific output format.
If the conditions for transmission are satisfied the program devhan
is called otherwise nothing happens.
4 funtou The timeout handler task (funtou) handles the system timeouts.
System timeouts are defined to perform given actions on an operational
record after a time interval starting from given events. Periodically the
Page 35
INDIAN AIRPORTS SKYWAY Administrator Manual
funtou task checks if any of the timeouts expired and eventually executes
the related job.
5 funhan The function handler task (funhan) is intended as the general interface
towards operational file for every kind of update operation.
It is activated by input operator’s procedure (add, mod, del) or by system
automatic functions (fillop, digital inputs, time-out handler …).
The managing of the record fields for updating or format and coherence
verification, performed according to the roules of the system, is tipical of this
level: every record must pass “through” this block to be put in the system
internal format.
In the record fields the following functions are performed:
format, coherence and presence checking on various fields
record status updating
automatic remarks or special fields generation
automatic blink field generation
modification and deletation on time-out base managing
automatic announcement requests sending
6 ffhand The free format handler task (ffhand) handles the free format messages
to be displayed on the free format network sending the information stored in
the free format database table.
7 lbhan It receives the subscriptions from the peripheral devices. If assigns the
device to a specific AREA SERVER and eventually to a VIRTUAL
CONTROLLER.
8 lbpoll It polls the AREA SERVERS to gets their STATUS (INS/OOS). It calls lbhan
if something change.
9 fillop The operational filler task (fillop) handles the filling of operational from
permanent online file.
The fillop task guarantees a DELTAHOURS (default) 24 hours time window
Page 36
INDIAN AIRPORTS SKYWAY Administrator Manual
of flights into the operational file.
10 perhan The permanent handler task (perhan) is intended as the general interface
towards permanent (online and offline) files for every kind of update
operation.
It is activated by input operator’s procedure (add, mod, del).
The managing of the record fields for updating or format and coherence
verification, performed according to the roules of the system, is tipical of this
level: every record must pass “through” this block to be put in the system
internal format.
11 devinz The device initialization handler task (devinz).
It performs the following function :
It sends to the devices the Layout according to the System
Configuration.
12 devhan The device handler task (devhan) constitues the software interface
between the Solari Information System and the network of different kinds of
displays connected.
It performs the following function :
to distribute all the informations to be displayed on the desired
peripherals;
to survey the operating status of the displays, and advise in real-
time the other components of SKYWAY System.
Page 37
INDIAN AIRPORTS SKYWAY Administrator Manual
5.2ASA SERVER
TTY TASK Feature
1 output Specific output from the ASA System.
2 asabook ASA book. Interface with the specific PAS SYSTEM.
3 asascd ASA scheduler
4 asadri ASA driver.
Page 38
INDIAN AIRPORTS SKYWAY Administrator Manual
6 SYSTEM MONITOR
For monitoring the system, Solari uses monit 4.9.
monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.
Monit is controlled via an easy to configure control file based on a free-format, token-oriented syntax. Monit logs to syslog or to its own log file and notifies you about error conditions via customizable alert messages. Monit can perform various TCP/IP network checks, protocol checks and can utilize SSL for such checks. Monit provides a http(s) interface and you may use a browser to access the monit program.
for more details visit the monit official web site www.tildeslash.com/monit.
A web interface with monit is available at http://lbsrv/tools/ For restarting a server simply select it, in related web page You will be able to “START SERVICE”, “STOP SERVICE” and “RESTART SERVICE”.
Page 39
INDIAN AIRPORTS SKYWAY Administrator Manual
As shown in the previous picture in case of MASTER L.B. Server all FIDS application will be up and running.
If the MONIT STATUS is checked on the SLAVE L.B. Server all the processes will have the stopped status.
To restart one application, for example tfthan, is only necessary to click on it (see next picture) and click on Restart Service button.
Page 40
INDIAN AIRPORTS SKYWAY Administrator Manual
As shown in the next picture the task tfthan will have the initializing status and, after 1 minute, will return to running status.
Page 41
INDIAN AIRPORTS SKYWAY Administrator Manual
7 LOGO INSTALLATION
All the monitors (TFTs) get the logos from the network via NFS.
Logos reside on the MASTER. The folder /export/incoming is available for the installation of new logos or for the logos updating. From a Windows workstation You have a read/write access to \\172.26.111.1\incoming. Use logo as username and password. Inside you’ll find the following folders (normally empty3):
FOLDER DESCRIPTION
CK_00 Put here the fullscreen images to be displayed on CHECKINS when closed. Name has to be: IMG01.jpg, IMG02.jpg, IMG03.jpg, IMG04.jpg, IMG05.jpg. Size should be 1024x576. (according with TFT resolution)
GT_00 Put here the fullscreen images to be displayed on GATES when closed. Name has to be: IMG01.jpg, IMG02.jpg, IMG03.jpg, IMG04.jpg, IMG05.jpg. Size should be 1024x576. (according with TFT resolution)
116x29 Put here the ailines logos to be dsplayed on TFT FLIGHTS MONITORS and Public Baggage Monitors. Filename is supposed to be <AIRLINE_CODE>4.jpg. Size should be 116x29. (according with TFT resolution)
1024x160 Put here the ailines logos to be dsplayed on TFT CHECKIN. Filename is supposed to be <AIRLINE_CODE>.jpg. Size should be 856x160. (according with TFT resolution)
420x101 Put here the ailines logos to be dsplayed on TFT GATES. Filename is supposed to be <AIRLINE_CODE>.jpg. Size should be 420x101. (according with TFT resolution)
When You disconnect5 from the service incoming the system executes a script6 that takes the logos from incoming, converts them to the appropriate size and format, and finally moves them in their final location.
3 Normally empty because the logos are moved in /export/logo and removed from /export/incoming.
4 <AIRLINE_CODE> from DESTINATIONS vocabulary.
5 Windows clients have an autoreconnect feature, so for disconnecting you should go in My Network Places, right click and choose “Disconnect Network Drive”. Otherwise the system will consider closed the connection after an inectivity of 5 minutes.
6 The shell script is /solids/script/logocvt, that logs in /solwrk/logocvt/YYYYMMDD.
Page 42
INDIAN AIRPORTS SKYWAY Administrator Manual
8 LOG FILES
All FIDS Application running on the L.B. Server will create an own log file.
The log files location is the directory: /solwrk.
Inside /solwrk directory there are many sub-directories, one for each FIDS application: the name of each sub-directory is the same name of the FIDS Application (see section 5).
Inside each sub-directory there will be 1 log file for each day. The log file name follows the rule:
YYYYMMDD
Example: we want to check the log file
/solwrk/funhan/20091014
(14/10/09 15:29:34) received from JAI@lapence 'OPEDEL' 'WHERE ID="116126"'
(14/10/09 15:29:34) SELECT * FROM OPEFIL WHERE ID='116126' : FOUND ID=116126 <SOLGREP>#116126#0#AD#G8#112#</SOLGREP>
(14/10/09 15:29:34) NotifyServers(DISDEL) : DISDEL ID="733238" ENABLED="1" EFF_OPEN_TIME="20091013.163226" NETWORKNAME="PUB_A
(14/10/09 15:29:34) NotifyServers(DISDEL) : DISDEL ID="733239" ENABLED="1" EFF_OPEN_TIME="20091013.163226" NETWORKNAME="PUB_A
(14/10/09 15:29:34) NotifyServers(DISDEL) : DISDEL ID="733240" ENABLED="1" EFF_OPEN_TIME="20091013.163226" NETWORKNAME="PUB_A
(14/10/09 15:29:34) NotifyServers(DISDEL) : DISDEL ID="733241" ENABLED="1" EFF_OPEN_TIME="20091013.163226" NETWORKNAME="LCD_A
(14/10/09 15:29:34) NotifyServers(DISDEL) : DISDEL ID="733237" ENABLED="1" EFF_OPEN_TIME="20091013.163226" NETWORKNAME="LCD_A
(14/10/09 15:29:34) NotifyServers(DISDEL) : DISDEL ID="733242" ENABLED="1" EFF_OPEN_TIME="20091013.163226" NETWORKNAME="OARR"
(14/10/09 15:29:34) mypid=6623 - replying JAI@lapence retcode=0
In the above-mentioned example the log file records between the 1st and the last row are referred to a deletion of a flight from the operational file.
Page 43
INDIAN AIRPORTS SKYWAY Administrator Manual
Please note that:
(14/10/09 15:29:34) received from JAI@lapence 'OPEDEL' 'WHERE ID="116126"'
JAI@lapence = USER (JAI), SERVER (lapence)
(14/10/09 15:29:34) SELECT * FROM OPEFIL WHERE ID='116126' : FOUND ID=116126 <SOLGREP>#116126#0#AD#G8#112#</SOLGREP>
FLIGHT CLASS (AD)
AIRLINE (G8)
FL.NUMBER (112)
(14/10/09 15:29:34) mypid=6623 - replying JAI@lapence retcode=0
retcode=0 OPERATION COMPLETED SUCCESSFULLY
Page 44
INDIAN AIRPORTS SKYWAY Administrator Manual
9 LINK WITH AFTN
In some airports (like NAGPUR) has been activated the link between FIDS and AFTN systems, through a serial port connection on the WEB Server.
This link is unidirectional (from AFTN to FIDS) and can be helpful to update the following fields:
- AIRCRAFT TYPE
- ESTIMATED TIME
- TOUCH DOWN TIME
- TAKE OFF TIME
In order to detect the link status and the received messages the following web page can be accessed:
http://lbsrv/tools
and then AFTN messages must be selected.
TOOLS web page.
Page 45
INDIAN AIRPORTS SKYWAY Administrator Manual
AFTN messages web page.
Page 46