26
1 Preparing for the Unexpected ITSM Conference – April 21, 2008 Steve Lipshetz – Senior Business Continuity Consultant
1
Preparing for the UnexpectedITSM Conference – April 21, 2008
Steve Lipshetz – Senior Business Continuity Consultant
2
Agenda
The Risk of a Disaster Business Continuity and Disaster Recovery 9/11 Changed Everything Where Do We Start? What is Business Resilience? Building a Partnership “Right-sizing” the Program Auditing and “Testing” the Program Looking Towards the Future Key Take Aways
3
The Risk of a Disaster
Business and systems operations face four categories of risks:
o Natural Disaster or Weather Relatedo Terrorismo Company Facility / Buildingo People
Low probability / high impact Certain risks more likely than others – Midwest / tornado
4
Business Continuity vs. Disaster Recovery
Business Continuity (Led by business area)o Company’s game plan for keeping your critical business operations
working if:• A company worksite is lost (permanent or temporary)• Access to computer systems and applications is lost or limited• The workforce is disrupted such as in a Pandemic
Disaster Recovery (Led by IT)o Company’s game plan for maintaining or restoring critical and non-critical
infrastructure, systems and applications
Joint Efforts – Business and ITo Assure that most critical business operations are “recovered” firsto Assure that critical systems in support of business are recovered first
5
9/11 Changed Everything
Many impacted businesses went out of businesso Lost datao Lost business expertise
Difficulty for other companies to get back in businesso Inadequate recovery planso Lost business expertise
CEO’s and Boards ask questions:o How would our Company fare?o Is our data safe?o Do we have adequate recovery plans?o Do people know what to do in a disaster situation?o Can we survive?
6
Where Do We Start?
Risk Evaluation and Controlo Identifying risks and potential riskso Identifying potential consequences if risk becomes reality
Business Impact Analysiso Identifying critical business processes and recovery time objectiveso Identifying dependencieso Identifying consequences of disruption
• Financial• Legal• Regulatory• Reputation• Personnel
7
Where Do We Start?
Business Process / Functions:
Assign Recovery Time Objectives (RTO’s)
People Performing process /
function
Assets and EquipmentNeeded to perform process /
function
SoftwareNeeded to perform process / function
Internal DependenciesOther departments that
department depends upon to perform process / function
External DependenciesThird parties that department
relies upon to perform process / function
Vital RecordsRequired to
perform process / function
Recovery Locations
Alternate location(s) for people / assets
External CustomersThird parties that rely upon
department to perform process / function
8
Where Do We Start?
DRI (Disaster Recovery Institute) Internationalo Ten professional practices for Business Continuity planners
NFPA 1600 Generally Accepted Practices for Business Continuity
Practitionerso Draft collaboration – Disaster Recovery Journal and DRII
Business Continuity Institute Good Practices Guidelineso Six areas for developing an effective Business Continuity program
9
Where Do We Start?
Coordination with External Agencieso NIMS - National Incident Management Systemo ICS – Incident Command Systemo Critical Incident Protocol Program
• Joint Public / Private partnership• Michigan State University / DHS grant
• Brown, Dane and Eau Claire Counties• Milwaukee and Racine
10
Where Do We Start?
Key element in building, implementing and maintaining an effective program, and executing plans in a disaster is……
11
Where Do We Start?
Effective and timely Communication!!!
12
What is Business Resilience?
Newest preparedness and planning philosophyo The ability to avoid, minimize, withstand and recover from the
affects of adversityo The ability of an organization to sustain the impact of a business
interruption and recover and resume its business operations in order to continue to provide an acceptable level of services
o All encompassing planning methodology• Business Continuity• Disaster Recovery• Crisis Management
13
What is Business Resilience?
Business Continuityo Company’s game plan for keeping your critical business operations
working if:• A company worksite is lost (permanent or temporary)• Access to computer systems and applications is lost or limited• The workforce is disrupted such as in a Pandemic
Disaster Recoveryo Company’s game plan for maintaining or restoring critical and non-critical
infrastructure, systems and applications
Crisis Managemento Intervention and coordination by individuals or teams before, during, and
after an event to resolve the crisis, minimize loss, and otherwise protect the organization
14
Building a Partnership
People + Systems and Data = Business Process executiono Business focal point and business department representativeso IT focal point and IT experts (infrastructure, systems, PCs, telephony)
Joint planning – all types of disruptions (worksite, system, people)o Criticality of business process drives system availability requirementso Business and IT plans must be in synco Protection of all electronic datao Paper vital records management
Joint testing of planso Business areas are dependent on IT for “business as usual”o Plans need to be reviewed and tested jointly to assure that business
processes can be maintained and/or restored following a disruption
15
Right-Sizing the Program
Generally accepted practices are the minimum of what should be doneo Latitude within what is implemented
Development + Testing plans = $$$$$o Cost of establishing disaster recovery for infrastructure and systems
• Network design• Alternate data center and equipment costs vs. vendor solution
o Cost of establishing worksite recovery for people and business processes• Strategies• Other company facilities
o Cost of establishing plans for loss of personnel Regulation / audit sets the bar for what is expected in certain
industrieso Financialo Insuranceo Health care
16
Auditing and Testing the Program
Business Continuity and Disaster Recovery Plan requirementso Must be complete!o Must be executable!
Plan review process should be joint with Audito Develop process including criteria for reviewo Develop review templateo Pilot with Audit and other selected groupso Develop schedule
17
Auditing and Testing the Program
Types of Drills and Exerciseso Calling tree - actual
• Tests process of contacting personnel• Assures that current contact information is correct
o Tabletop exercise (structured walkthrough of plan) - simulation• Disaster scenario given to facilitator• Department personnel talk through what they would do and reference
their plans• Could be designed to exercise any type of plan• Most knowledgeable people can be “sent on vacation”!
18
Auditing and Testing the Program
Types of Drills and Exerciseso Disaster recovery exercises - actual
• Led by IT• Business area testing involvement• Joint follow-up meeting and “lessons learned” document• Tasks are assigned and completion is tracked
o Crisis management drills – actual and simulation• Contact crisis management team members• Should ideally be a “surprise”• Use of the Emergency Operations Center• Walk through a scenario• Optional to involve others not in the room, but do not execute any plans
19
Auditing and Testing the Program
Types of Drills and Exerciseso Worksite recovery exercise - actual
• Led by business area• Significant IT involvement• Selected business groups go to designated recovery site and work• Tests both business and IT processes in support of the business• Joint follow-up meeting and “lessons learned” document• Tasks are assigned and completion is tracked
20
Auditing and Testing the Program
Types of Drills and Exerciseso Scenario-based drills
• Considerable planning needed• Core planning team• Involves many different business areas and processes• Could involve one or multiple simultaneous scenarios• People talk through what they would do and contact others as needed• Plans are not executed• Joint follow-up meeting of core team and “lessons learned” document• Tasks are assigned and completion is tracked
21
Looking Towards the Future
Most recent threat – Avian Fluo Plans adequately covered loss of worksite or loss of systemso Major loss of personnel was never consideredo Pandemic situations re-occur – if not this threat, what next?
Terrorist attacks – are they inevitable?o What will be targeted?
• Population hub• Symbol of the United States• Transportation• Electric or natural gas infrastructure• Water supply
22
Looking Towards the Future
H.R. 1/ Public Law 110-53: Implementing Recommendations of the 9/11 Commission Act of 2007
o Signed into law August 3, 2007o Most sections of the law relate to government and public entitieso Two sections relate to private sector, but are not mandatory
• Strengthening the use of the Incident Command System by coordinating with private industry to promote preparedness
• Private sector preparedness including certification guidelines and standards
o Are we one terrorist attack away from mandatory requirements?• In critical industries?• In all industries?
23
Key Take Aways
Protect your data! Develop plans to re-build your technical environment Business Continuity Planning
o Something is better than nothingo Senior Executive buy-ino If in a regulated industry, meet all federal and state regulatory requirementso If not regulated:
• How best can committed $$$$$ be spent• Work with “critical” business processes and departments first
24
Key Take Aways
Develop Business / IT partnership approach to planningo Execution of any plans requires both areaso Coordination of planning and testing will help keep chaos manageable
Test, test, test, test, test………………………o You never know how good a plan is until you put it to a testo Problems in testing are good – you can remediate the problem!o If you have no problems, was the test designed properly?
Communicateo Clear and conciseo To / from all levels of the organizationo To / from all departments with which you have dependencies o To / from all critical 3rd parties
25
Key Take Aways
"Above all else, we certainly know one thing from past such events: preparation makes all the difference. Although events never unfold exactly as we have planned, having no plan is simply a plan for failure.“
Kerry Killinger – Chairman and CEO of Washington Mutual Inc
