BBusiness Resumption From An Application Perspectiveusiness Resumption From An Application Perspective
Presented By:Presented By:Steven J. Skidmore Steven J. Skidmore IT Director IT Director Martin Clearwater & Bell LLPMartin Clearwater & Bell LLP
Thomas Huson Thomas Huson Director, Professional ServicesDirector, Professional ServicesAdvanced Legal Systems, Inc.Advanced Legal Systems, Inc.
Lawnet 2003Lawnet 2003Networking & EducationNetworking & EducationApplications TrackApplications TrackAugust 21, 2003 10:00amAugust 21, 2003 10:00am
Session OverviewSession Overview
The session is based on a worst case scenario The session is based on a worst case scenario Total devastation to computer room facilitiesTotal devastation to computer room facilities
Core components of sessionCore components of session
– Decision ProcessDecision Process– Build Build – RebuildRebuild
Disaster RecoveryDisaster Recovery
Disaster Recovery – The process of Disaster Recovery – The process of recovering from an event that has recovering from an event that has negatively affected the operation of an negatively affected the operation of an organizationorganization
– Disaster has occurredDisaster has occurred– Management declares disasterManagement declares disaster– Activate disaster recovery planActivate disaster recovery plan
Key Players Key Players
The FirmThe Firm– Litigation Firm Litigation Firm
Main Office Located Midtown ManhattanMain Office Located Midtown Manhattan Two Satellite OfficesTwo Satellite Offices Medical Malpractice DefenseMedical Malpractice Defense 80 Attorneys80 Attorneys
– IT DepartmentIT Department IT DirectorIT Director 2 Support Staff Employees2 Support Staff Employees
– Consulting TeamConsulting Team Sr. EngineerSr. Engineer
Case Study ObjectiveCase Study Objective
To find and implement a disaster To find and implement a disaster recovery solution that will restore the recovery solution that will restore the Firm’s critical business applications Firm’s critical business applications within 72 hours.within 72 hours.
The Decision ProcessThe Decision Process
Assessing Our RiskAssessing Our Risk
– Changed Attitudes –Everything has Changed Attitudes –Everything has changed since 9/11.changed since 9/11.
– Could not afford not to Could not afford not to Interruption to business can be costlyInterruption to business can be costly
– Responsibility to clientsResponsibility to clients– Our location puts us at a high riskOur location puts us at a high risk
United Nations, Israeli Consulate, United Nations, Israeli Consulate, Grand Central StationGrand Central Station
– Anticipate the worst case scenarioAnticipate the worst case scenario
Factors determining a DR solutionFactors determining a DR solution
– Cost FactorCost Factor Had a fixed budget to deal withHad a fixed budget to deal with
– Comdisco Recovery CenterComdisco Recovery Center– Work with Consulting team and keep Work with Consulting team and keep
project in house.project in house. Utilize satellite officeUtilize satellite office
The Decision ProcessThe Decision Process
Time FactorTime Factor– Identify critical business processesIdentify critical business processes– How fast did we need to be operationalHow fast did we need to be operational
Connection FactorConnection Factor– Who was going to access the recovery Who was going to access the recovery
system?system? Identify Tier levelsIdentify Tier levels
– Where are they going to access from?Where are they going to access from? OnsiteOnsite OffsiteOffsite
– How were they access the system?How were they access the system? CitrixCitrix
The Decision ProcessThe Decision Process
Application FactorApplication Factor– Essential Applications (0-72hrs) = Essential Applications (0-72hrs) = Disruption in Disruption in
service exceeding 72 hours would service exceeding 72 hours would jeopardize the operation of the Firmjeopardize the operation of the Firm
EliteElite– Time / BillingTime / Billing– Case Management / Calendar Docket SystemCase Management / Calendar Docket System
MSExhangeMSExhange– EmailEmail– CalendarCalendar– ContactsContacts
iManage / WordPerfectiManage / WordPerfect– Document ManagementDocument Management– Word ProcessingWord Processing
The Decision ProcessThe Decision Process
Recommended Applications (72hrs – 1wk) = Recommended Applications (72hrs – 1wk) = Disruption in service exceeding 1 week would Disruption in service exceeding 1 week would jeopardize the operation of the Firmjeopardize the operation of the Firm
– Microsoft Office SuiteMicrosoft Office Suite– SummationSummation– Record AccessRecord Access– CD-Rom ApplicationsCD-Rom Applications– Internet AccessInternet Access
The Decision ProcessThe Decision Process
Non-Essential Applications ( > 1 week)Non-Essential Applications ( > 1 week)
– Nice to have – but would not Nice to have – but would not jeopardize the operation of the Firm if jeopardize the operation of the Firm if they were missingthey were missing
The Decision ProcessThe Decision Process
Decision Process ResultDecision Process Result
Warm SiteWarm Site– Long Island Satellite OfficeLong Island Satellite Office– T1 ConnectionT1 Connection– Off Site Backup Tape Storage FacilityOff Site Backup Tape Storage Facility– Employees who live 15mins awayEmployees who live 15mins away
Designing a Recovery Designing a Recovery EnvironmentEnvironment
Application Functionality in the Live Application Functionality in the Live Environment Spans 9 servers in 2 DomainsEnvironment Spans 9 servers in 2 Domains Compress basic functionality into other Compress basic functionality into other hardware at minimal costhardware at minimal cost The Mission Critical apps the Firm The Mission Critical apps the Firm Identified not conducive to simple tape Identified not conducive to simple tape restore. Specifically:restore. Specifically:
– ExchangeExchange– SQLSQL– Elite’s InformixElite’s Informix
Challenges:
Building a Recovery Warm SiteBuilding a Recovery Warm Site
Communication Redundancy Communication Redundancy Data Replication SchemeData Replication Scheme Active Directory ConfigurationActive Directory Configuration Solid Hardware at Recovery SiteSolid Hardware at Recovery Site New Software InstallationsNew Software Installations
Solution
Building a Recovery Warm SiteBuilding a Recovery Warm Site
Communication configurationCommunication configuration– T-1 frame between sitesT-1 frame between sites– T-1 at recovery siteT-1 at recovery site– Redundant linkRedundant link– Recovery site has own scope of valid Recovery site has own scope of valid ip’s ip’s
Hardware configurationHardware configuration– Single high performance server to Single high performance server to house Majority of appshouse Majority of apps– Additional “light-weight” box for 2Additional “light-weight” box for 2ndnd domaindomain
Solution
Building a Recovery Warm SiteBuilding a Recovery Warm Site
Software ConfigurationSoftware Configuration– New install of mission critical software New install of mission critical software from factory mediafrom factory media
Server/Active Directory (Into existing forest)Server/Active Directory (Into existing forest) CitrixCitrix MS ExchangeMS Exchange SQL/iManageSQL/iManage RSASecureIDRSASecureID EliteElite ArcserveArcserve
–Installed, Tested – Services disabledInstalled, Tested – Services disabled
Solution:
Building a Recovery Warm SiteBuilding a Recovery Warm Site
ReplicationReplication– Arcserve copy utilityArcserve copy utility
Runs every 4 hoursRuns every 4 hours Copies over all files with archive bit flaggedCopies over all files with archive bit flagged Resets flagResets flag
Active Directory ConfigurationActive Directory Configuration– Disaster Recovery OUDisaster Recovery OU– Security Group “Recovery”Security Group “Recovery”
For Different application pointers, drive mappings For Different application pointers, drive mappings etc. etc.
Solution:
Rebuilding the EnterpriseRebuilding the Enterprise
AssumptionsAssumptions– Catastrophic EventCatastrophic Event– May not Have Access to the Most May not Have Access to the Most Recent Backup SetRecent Backup Set– Localized EventLocalized Event– We Utilize a Full Backup Strategy We Utilize a Full Backup Strategy
Rebuild ProcessRebuild Process– SharesShares– SQLSQL– Elite InformixElite Informix– MS ExchangeMS Exchange
Rebuilding the EnterpriseRebuilding the Enterprise
Obtain Most Current Backup SetObtain Most Current Backup Set– May be 48 Hrs oldMay be 48 Hrs old
Begin Restore Process to Bring Recovery Begin Restore Process to Bring Recovery Environment OnlineEnvironment Online
– Restore SharesRestore Shares Merge TapeMerge Tape “ “Replace newer files only”Replace newer files only” (Should not actually lay down any data)(Should not actually lay down any data) AppsApps DataData DocsDocs HomeHome ShareShareStartMenusStartMenus
Rebuilding the EnterpriseRebuilding the Enterprise
Restore SQL Database for iManageRestore SQL Database for iManage– Merge SQL Backup tapeMerge SQL Backup tape– Restore to . . .\SQLBACKUP directoryRestore to . . .\SQLBACKUP directory– Note Backup (BAK) fileNote Backup (BAK) file– Note Transaction log files Newer than backup fileNote Transaction log files Newer than backup file– Restore Database in SQL Enterprise ManagerRestore Database in SQL Enterprise Manager
“ “Restore database by device”Restore database by device” Select “Force restore over existing databaseSelect “Force restore over existing database Select “Leave Database nonoperational but able to Select “Leave Database nonoperational but able to restore logs”restore logs”
– Apply Transaction Log Files in SQL Enterprise ManagerApply Transaction Log Files in SQL Enterprise Manager Must be done individually if more than oneMust be done individually if more than one Select “Force restore over existing database”Select “Force restore over existing database” When all Log Files Applied Set Database to Online ModeWhen all Log Files Applied Set Database to Online Mode Modify directory pointers in Database (File server/Index Modify directory pointers in Database (File server/Index server)server) Change application pointers in iManageChange application pointers in iManage Start iManage ServicesStart iManage Services
Rebuilding the EnterpriseRebuilding the Enterprise
Rebuilding the EnterpriseRebuilding the Enterprise
Rebuilding the EnterpriseRebuilding the Enterprise
Informix database for EliteInformix database for Elite– Restore from tape via command Restore from tape via command line in KORN Shellline in KORN Shell– Does Not Involve ARCServeDoes Not Involve ARCServe
Onstat – LOnstat – L Onstat - KOnstat - K Ontape – ROntape – R Onmode – MOnmode – M Onstat – L (To Verify Online Status)Onstat – L (To Verify Online Status)
Rebuilding the EnterpriseRebuilding the Enterprise
Microsoft ExchangeMicrosoft Exchange– Merge TapeMerge Tape– Start Exchange ServicesStart Exchange Services– Modify User Accounts via ADSIModify User Accounts via ADSI
Launch ADSI Edit Application and CLEAR Launch ADSI Edit Application and CLEAR for Each User:for Each User:
– homeMDBhomeMDB– homeMTAhomeMTA– mailNicknamemailNickname– All other msExch propertiesAll other msExch properties
Rebuilding the EnterpriseRebuilding the Enterprise
Rebuilding the EnterpriseRebuilding the Enterprise
Rebuilding the EnterpriseRebuilding the Enterprise
Microsoft Exchange (Cont. . .)Microsoft Exchange (Cont. . .)– Create New Mailbox in AD Users and Create New Mailbox in AD Users and computerscomputers– Send test mail to activate mailboxSend test mail to activate mailbox– Brick level restore to new mailboxesBrick level restore to new mailboxes– Move user account into disaster Move user account into disaster recovery OUrecovery OU– Add account to “recovery” groupAdd account to “recovery” group
ConclusionConclusion
Disaster Recovery SolutionDisaster Recovery Solution– DR solution needs to be tailored to the DR solution needs to be tailored to the
Firm’s size, resources and geographic Firm’s size, resources and geographic locationlocation
– Anticipate worst case scenarioAnticipate worst case scenario– Buy as much as you canBuy as much as you can– Documentation is keyDocumentation is key– Stay honestStay honest– UpdateUpdate