Disaster Recovery,Business Continuity Plan

& Backups

Justin Broton4th February 2010

Government of GibraltarInformation Technology & Logistics Department

Produced by: Justin Broton Date: February 2010 (Slide 2)

Background Information on Interreg EU Projects

Admitron I (Infrastructure)

Admitron II (Corporate Services)

Provided interconnectivity to all Government major sites with a Private Fibre Optic backbone.

Created a Government Secure Intranet (GSI).

All remote sites linked by one standard 100 Mb Optic Fibre.

Central Administration & DR/Backup Sites are both links of 1Gb Optic Fibre.

Internal & External Mail.Public Web-Publishing.Internal Web-PublishingContent & Email filtering.

Centralised Management & Troubleshooting.Resource Sharing.Basic Backup & Storage.

Produced by: Justin Broton Date: February 2010 (Slide 3)

Consequences of Government Secure Intranet on IT&LD

Legal Compliance

Backups

Service Level Agreement

EU & Minister Initiatives

Information SharingThe Intranet has created a higher degree of responsibility for the IT department to maintain systems.

Going Green

IT is now responsible for an Increased availability of services to be continually online / active.

Elevated IT responsibility to backup core systems and more information requires backing up.

eDiscovery and anti-tampering legal compliance of data retention (7 years).

As a direct result more emails & eDocuments are being used / transferred instead of using traditional printed paper.

Comply with Regulatory standards based on those of the EU, UK & Gibraltar.

Produced by: Justin Broton Date: February 2010 (Slide 4)

Software SLA for Government departmentsMotor Vehicle Licensing & Driver System.Public Service Vehicle Application.CSRO (ID Cards).Non EU Citizens Register.Production of Birth, Death, Marriage Certificates.No6 Registry.Magistrates Court.Fines Application.Supreme Court Application.Prison Sentence Application.Moroccan Visa Waiver Application.GHA Salaries.Treasury Salaries.Treasury Wages.Treasury Pensions.Income Tax.Social Insurance Contributions.Social Insurance Cashier Application.TAX Exempt Application.Statistics Application.Housing Application.GMA Ship Registry Application.VB Management Scripts.Logon Scripts.Screen Saver. Internal Government Telephone Directory Service.

Maintenance of over 26+ Applications and growing

Product Support

MaintenanceEnhancements

Emergency FixesApplication ContinuityApplication MonitoringDB Monitoring

Regulatory ChangesMinor EnhancementsCorrective MaintenanceDB AdministrationSecurity & Code Review

Produced by: Justin Broton Date: February 2010 (Slide 5)

Upcoming IT&LD ProjectsIntranet Based SystemsDepartment of Transport

Roadworthiness Tests (MOT) Booking and on line payment.Driving Test Booking and on line payment.Learners Driving Licence.Automatic Notification to clients on renewal dates.Help Desk for Queries & Complaints.

Civil Status & Registration OfficeBirth, Death & Marriage Certificate order and online payment.Gibraltarian Status and Other Certificates.Help Desk for Queries & Complaints.

No6Subscription Service for the download and payment of Gazettes online.

Income Tax/Social InsuranceAdvice clients (PAYE, Self Employed or Corporate) that an assessment has been issued.Online Tax Returns.Online P8s, P7As etc…Online Claims for all allowances, Marriage, Child, Mortgage, Life Insurance etc…Online payment of Assessments, SI and Agreements.Client can check SI contributions paid by employer on his/her behalfHelp Desk for Queries & Complaints.

CustomsThe new Customs Application (ASYCUDA World) will allow for electronic interaction with clients.

EducationScholarship Grant Application.Student Loan payment online.School and Nursery Enrolment.Help Desk for Queries & Complaints.

HousingOnline payment of rent.Help Desk for Queries & Complaints.

Non Intranet Based SystemsJudiciary

Online payment of parking tickets. Online payment of Fines. Help Desk for Queries & Complaints.

Port AuthorityTonnage Dues.Berthing Charges.Small Boat Mooring Fees.Port Arrival & Departure Passenger Tax.Port Operator & Harbour Craft Licence Fees.Bunkering Charges.Misc. Charges.Help Desk for Queries & Complaints.

Project Est. Duration Est. Staff RequiredCustoms Asycuda (Started 04/2009) 24 Months 4 (2 Customs)Vehicle Registration Disk 10 Months 2Motor Vehicle Insurance 12 Months 2Tachograph Card (Started 06/2008) 18 Months 1New Driving Licence 08 Months 1International Driving Permits 02 Months 1Old Age Pensions (Started 03/2009) 10 Months 2Social Security 18 Months 2EESSI (EU Application) 15 Months 2CSRO 12 Months 1

Inundation for requests of new applications. All these are required due to the implementation of EU Directives or new

Government Initiatives from different Ministers.

Keeping up with EU Legislation / regulatory standards, will in most cases involve upgrading current systems.

Produced by: Justin Broton Date: February 2010 (Slide 6)

Infrastructure as current

Produced by: Justin Broton Date: February 2010 (Slide 7)

Archiving for Legal Compliance (7 years)

St. Jagos (offsite backup site)Library Street (main site)

Year 1 (Jan – Jun – Dec)Year 2…..Year 3 …..Year 4 …..Year 5 …..Year 6 ….. Year 7 (Jan – Jun – Dec)

PASSWORD (256 BIT)ENCRYPTED (256 BIT)WORM

AS400Income Tax, DSS, Treasury…

PST Bimonthly Retention (Manual Process)

EMAIL & USER/PERMISSIONS SERVER

Real-Time Replication & Previous VersionsNo Single Instance (20% to 70% saving)

EXTRA NOTES:EU Compliance of data retention, data protection & no tampering legal requirements, no spare servers available for acceptable SLA disaster

recovery. Single point of failure on some backups. Training requirements for ITLD & users. R&D virtually inexistent. No budget for backups. Reactive approach. Encryption adds extra burden on disk

requirements. Extra tape management.

Existing layout does not include external departments not in Intranet as of yet. No auditing or monitoring/maintenance software available e.g.

growth of data, de-duplication, Intelligent reclaiming of disk space, maintaining exclusion lists (these are all manual interventions). No continuous protection of emails/data until evening backup (always

potential of 1 day loss of emails/data).

Time to deal with recording & correcting potential problems.

There are no desktop or laptop backups as at present, important files & emails saved can be lost in event of equipment failure.

DATABASES & INTRANET(MVTC, CSRO, DSS Records, Registry, Intranet….)

Daily

Real-Time Sync Copy

FULL SERVER BACKUPS FOR QUICK DISASTER RECOVERY

(Printer, Internet Access, Domain Controllers….)

33+ Servers in Intranet

ALL DEPT DATA

Archiving for Legal Compliance (7 years)

160+ / 205+ Gigabytes800+ Users

367+ Gigabytes14+ Databases

B2D Daily + Weekly + Monthly

Daily (Granular per Mailbox & User Login)

None at present

Daily

Real-time Synchronisation

PST Outlook User Archive

510+ Gigabytes1,057,255+ Files86,855+ Folders

Backups as of present

Produced by: Justin Broton Date: February 2010 (Slide 8)

Looking ahead on Interreg EU Project

Disaster Recovery, Business Continuity Plans & BackupsFundamentals of the Project.Obtaining Management / Government / EU Commitment.Provide Network Resilience.Develop Disaster Recovery Data Centre.

Physical Security of Data Centre’s.Secure Access to IT&LD & Offsite Data Centre’s.

Identify Critical Department Data/Applications.Departmental own Contingency / Continuity Plans.

Procure Hardware & Backup/Replication Software for Main site & Disaster Recovery Data Centre.Disaster Recovery Prevention.Offsite DR (different country / different tectonic plate).

Produced by: Justin Broton Date: February 2010 (Slide 9)

Fundamentals of the project

What are we aiming to be protected against?Hardware, software failures or insertion of malicious code.

Telecommunication breakdown or disruptions.

Power failures or instabilities.

Environmental concerns such as smoke, fire, explosions, floods, building structural problems and earthquakes.

Sabotage, terrorism & public disorders.

What are the benefits?Providing a sense of security.

Minimizing risk of delays.

Guaranteeing the reliability of standby systems.

Minimizing the chances of data loss.

Improved performance and efficiency at a lower cost.

Produced by: Justin Broton Date: February 2010 (Slide 10)

Obtaining Management/Government/EU Commitment

Top management in all areas must support and be involved in the development of the disaster recovery planning process. Therefore in terms of personnel and financial resources, the tasks and procedures detailed in the plan should represent their commitment to response, resumption, recovery and restoration planning.

At all times it must be seen we are applying a proactive approach to disaster recovery to reduce data loss mitigation.

Departmental Contingency Plans must be in place in the event of a potential Disaster Recovery scenario.

IT Disaster Recovery Planning/Solutions & IT as a whole should be seen as an investment and not as an expense.

Produced by: Justin Broton Date: February 2010 (Slide 11)

Infrastructure as proposed (Network Resilience)

Produced by: Justin Broton Date: February 2010 (Slide 12)

Develop DR Data Centre

Power UPS

Power – Emergency Generator Power

Structured Wiring

Racks and Accessories

Air Conditioning Units

Management and Monitoring (Nagios, Spook – Early Prevention System)

Fire Detection and Extinguishing

Building construction and adaptation works on already existing structure eg. Raised flooring

Power – Electrical structure

Provide state of the art technologies in different fields of Facilities Infrastructure in the following areas:

Produced by: Justin Broton Date: February 2010 (Slide 13)

Physical Security of DR Centre’s

Access Control

Video Surveillance

Secure Access to IT&LD & Offsite Data Centre’s

Provide state of the art technologies in different fields of Facilities Infrastructure in the following areas:

Produced by: Justin Broton Date: February 2010 (Slide 14)

Identify Critical Department Data/Applications Impact / Risk Analysis & AssessmentDepartments must identify the amount of time they can remain inactive, whether in part of or as a whole department.

Departments must identify what the amount of data they are able to lose.

Departments must identify what dependencies they have with other departments / organisations.

Departments must consider keeping vital records in both hard copy and electronic format (both offsite).

Departmental own Contingency / Continuity Plans Computers are downDepartments must identify what events, procedures can be carried out manually in the event of system unavailability.

A simple example are Public counters; these must be prepared to accept at least payments from the public to pay their motor vehicle fees & licenses, housing bills, tax fees, etc…

Produced by: Justin Broton Date: February 2010 (Slide 15)

Procure Hardware & Backup/Replication Software for Main Site & Disaster Recovery Data Centre

Blade Systems

Backup Software

SAN (Shared Area Network)

Replication

So there are no bottlenecks and to provide fast replication speeds.

Fibre Channel switches

Blade systems have a smaller footprint and are extremely green solutions due to their shared power and cooling capabilities. They also come with high efficiency power supplies. Today’s blades can achieve 92% efficiency.

VRaid, Hardware Level “Block” Single Instance (known as Deduplication).

VirtualisationTo maximize usage of servers we must continue to expand on virtualization.

Dependant on Supplier some have out of the box replication others might require purchasing.

File/Email/DB Single Instance & Enterprise solutions should only be considered.

Produced by: Justin Broton Date: February 2010 (Slide 16)

New Proposed system using Blade – SAN’s - VLibraries

BackupVirtual Library System

Tape Library

SAN

EmailData

Direct Access Storage

Applications, Printers & Domains Blade Chasis

VRaid

consolidation

replication

Produced by: Justin Broton Date: February 2010 (Slide 17)

Proposed Outcome

FC SITE A

DR/Backup Site

Virtual Library System

Tape Autoloader / Tape Drives

FC NETWORK

EXCH DDBB

SQL DDBB

DATA

VMWARE

Symantec BE Server

Symantec NetBackup

Server

Symantec Enterprize Vault

Server

FC SITE B

SAN

SAN

Produced by: Justin Broton Date: February 2010 (Slide 18)

Adding disks to a Direct Access Storage Server

R1

SP

1. Add disks.2. Configure the group of disks RAID.3. Distribute Data RAID 1.4. Configure more RAID disk groups and spare.5. Distribute data… start all over.

Traditional Disk

Storage Server (DAS)

– adding capacity /

disks

Legend

RAID 5 disk group

RAID 1 group

Unused capacity

Data

moving away from DAS servers

Produced by: Justin Broton Date: February 2010 (Slide 19)

Automatic Disk Installation using SAN

Enterprise Virtual

Array – Installation

1. The SAN creates a group of disks automatically known as a pool of disks.

2. The client decides on the capacity required for Vraid1 or Vraid5.

3. The SAN unit distributes the data – all the data and unused space is distributed throughout the groups of disks.

Produced by: Justin Broton Date: February 2010 (Slide 20)

Automatic Disk Pool Growth is ideal

Enterprise Virtual

Array – adding

capacity

2. Existing group and added group are seamlessly merged and data is distributed accordingly throughout all the disks.

1. Add new group of disks.

Produced by: Justin Broton Date: February 2010 (Slide 21)

What is Deduplication (Single Instance)?

• A technique that compares blocks of data already written to the backup hardware.

• If duplicate blocks are found a pointer is added to the original data and like that you avoid copying an exact block of data again on the backup hardware.

• This technique can be done through comparing blocks or files and it depends what sort of applications/data you are backing up eg. Databases compared to Office documents.

• There is no loss of data and its transparent to the user.

• There are exponential cost saving benefits in using Single Instance technology at all levels (file or block).

After Deduplication

After

Before DeduplicationOriginal Backup

After

Original Backup

Produced by: Justin Broton Date: February 2010 (Slide 22)

Disaster Recovery Prevention

Security, Monitoring & MaintenanceHardware & Software can cut down on potential DR threats such as

hardware, software failures inserted through malicious code.

Monitoring software must be in place for early detection of all types of

possible Hardware/Software failures. This includes Auditing of files, logs

which can help prevent loss of data through inexperienced users and can

lead to great amounts of data loss. It can also help identify and trace

potential malicious attacks on the system.

Produced by: Justin Broton Date: February 2010 (Slide 23)

Offsite Data held at different country preferably on a different tectonic plate

Simple, Cost Effective & Secure (encrypted)

Via an Internet Connection

World Class Secure at “The Docks” Docklands, London (UK)

Server running costs are very low £150-£250 a month.

One off cost of a single server with a good sized NAS Unit forsingle instance data storage.

Only to transfer critically assessed departmental data / applications.

Only to transfer the latest working copy of our critical servers.

Currently thanks to our partnership with Gibtelecom (ISP) we have the availability of using a high connection for transferral of data over the Internet.

Thank you for your time and please do not hesitate to ask any questions.

Justin Broton (MBCS) (BEng)IT Officerjustin.broton@gibraltar.gov.gi

Government of GibraltarInformation Technology & Logistics Department