Transition of Disaster Recovery
(DR) Planning Practices
DRJ Spring World 2010March 23, 2010
Randall J. Till, MBCPBCM Group
Business Continuity Program
Program Governance
Continuity Planning
Business Continuity Planning
Disaster Recovery Planning
Emergency Management
Today’s Focus
Program Management and Administration
g Focus
Agenda
Meeting Objective: Discuss how DR planning practices continue to transition based on b i d d h i h lbusiness needs and changing technology.
Agenda:• History: Advancements in DR Practices• Challenges: Economic Downturn• Opportunities: New Technology and DR Planning• Strategies: DR Planning ApproachStrategies: DR Planning Approach• Final Thoughts
March 23, 20102
History: Advancements in DR Planning
Technology Advancements
Disaster Recovery Enhancements
• Increased network speeds/bandwidth
• Data mirroring and replication practices
• Reductions in hardware costs/increases in
• Improved data backup and protection methods
• Reduction in recovery point objectives (RPO’s)
• Improved recovery time objectives (RTO’s)
3
costs/increases in capacity
• Internal recovery solutions
• Enhanced recovery strategies and testing options
History: High Availability (HA) Concepts -- DR Planning
Primary Site Alternate Site
BANKNET BANKNET
Shared
Disk
BANKNET BANKNET
Shared
Di k
Replication
Process
4
Administrative
Support Applications
Disk
Administrative
Support Applications
History: High Availability Concepts Impacts on DR
Primary SiteTypes of Recovery:
1 Node Failover Alternate Site2 Cluster Failover
BANKNET BANKNET
Shared
Disk
3 Site FailoverBANKNET BANKNET
Shared
Replication
Process
5
Disk
Administrative
Support Applications Administrative
Support Applications
Disk
History: HA Concepts Integrated with DR
Primary SiteAlternate Site
BANKNET
Shared Shared
Disk
BANKNET
BANKNET
Shared Shared
BANKNET
Replication
Process
6
Administrative
Support Applications
Disk
Administrative
Support Applications
History: Disaster Recovery Planning
Advancements:• Internal Recovery Solutions –
multiple data processing centers
Benefits:• Protection of data – no or limited
loss of information• Data protection methods (e.g.,
mirroring, replication, acceptable data loss)
• Advanced processing and recovery techniques (e.g., high availability, co-processing, parallel processing)
• Live switches to alternate sites (upgrades, planned outages, testing)
• Reduced Recovery Time Objectives (RTO’s = Immediate)
• Improved change controls to support operational resiliency
• Better ROI on DR investments
7
Limitations:• Limited Funding – all systems are not equal (Tiered Recovery Strategy)• False Validation - Live switch of a single system is not the same as full site
recovery
Challenges: Economic Downturn
Maintain Assess
DR Planning Cycle Challenges:• Do more with less
PrioritizeTest/Train
Maintain
Planning
Cycle
Assess Do more with less• Reduce costs & budgets• Reduce size & scope of
DR exercises• Maintain DR readiness
& plans with less• Keep personnel ready to
executeApprove
Plan
Implement
8
execute • Continue to address
changing business needs
Challenges: Pressure on BC Program
• Modify BC practices to gain efficiencies by streamlining planning efforts R l t b i i t d f di• Re-evaluate business requirements and funding
• Evaluate oversight on BC priorities and funding requests• Focus on most critical areas and risk factors to define
planning requirements• Migrate the BC function towards more of a central office
oversight role • Restructure testing and exercise strategies and
processes
9
Challenges: Adjust BC Approach
• Migrate BC department towards more of a central oversight role
R d fi BC i t t d d d d li bl– Redefine BC requirements, standards and deliverables
– Provide more of a oversight role with less facilitation
– Conduct training sessions to guide planning and prepare users
• BC Plans are owned and managed by the business areas – Business Area Coordinators responsible for ensuring BC
deliverables are achieved
– Business owners are responsible for readiness of BC Plans including maintenance, validation and sign-off
– New recovery requirements and systems funded by the business owners
10
Challenges: BC ProgramOwnership and Coordination
Corporate Headquarters Europe Asia Pacific Latin America
Coordination
Ownership IT Headquarters
Regional
Marketing
Sales
CustomerServices
HR
Finance
FacilitationEmergency, Business, Systems
Planners
Emergency, Business
Emergency Plans
Division Coordinators
(Primary)
Coordinators
(Secondary)
Keys:• Distribute planning
efforts to the business areas
Legal
IT
Business Continuity Plans
Business, Systems Planners BC
PlansDR
PlansIT Change
Management
areas • Establishes a
Headquarter planning approach
• Independent update and validation of plans
Opportunities: New Technology Challenges Facing Organizations
• Budget constraints – find ways to reduce costs!• Hardware continues to grow in the IT environment and DRHardware continues to grow in the IT environment and DR
requirements has a doubling effect• One-for-one hardware setup in the recovery environment
is expensive and not effective • Software and maintenance costs continue to grow as
individual servers are setup for recovery systems• Recovery environment is difficult to maintain due to
12
• Recovery environment is difficult to maintain due to frequent production changes
Opportunities: New Technology
• Virtualization and Cloud Computing technologies will have a significant impact on DR planning practices
• Data backup/recover strategies will continue to migrate from tape to replication processes for critical systems
• DR hardware requirements will be reduced through the consolidation of applications on virtual servers
• DR plans and procedures will be simplified with improved tools and procedures to switch applications
13
• New outsourced DR services could evolve at a more cost effective price
Opportunities: Virtualization Concept
Primary Site
ABC
DEF
Alternate Site
ABC
DEF
GHI
JKL
MNO
GHI
JKL
MNO
Benefits:
Data Base
Security
Software
Application
ABC
14
• Systems consolidation reduces hardware demands
• Better utilization of hardware
• Reduction in utilities (green)
• Reduction in software fees
Opportunities: Cloud Computing
Primary Site
ABC
DEF
Alternate Site
ABC
DEF
SAN
15
GHI
JKL
MNO
GHI
JKL
MNO
Opportunities: New Technologies – DR Benefits
• Improved data backup capabilities with virtualization, better RPO’s with less loss of data
• Reduction in hardware costs providing a better business case for DR plans
• Simplification of DR plans and procedures with improved tools and procedures to switch applications
• Improved operational resiliency by switching systems to l d t d t h l d
16
manage planned outages and technology upgrades
• Enhanced Outsourced DR Services expected
Opportunities: New Technology –DR Concerns
• Disruption to a virtual server impacts multiple business systems
• Implementing the necessary tools, processes and training to effectively manage new technologies governanceeffectively manage new technologies - governance
• Tracking and managing the distribution of applications over a virtualized cloud computing environment
• Maintaining sufficient capacity to support a large processing outage
• Ability to validate recovery plans in a live virtualized cloud
17
computing environment
• Maintaining recovery plans & strategies for less critical systems
• Enhancing systems to take advantage of new technology.
Strategies: Redesign of DR Planning Practices
• DR Planning practices have evolved over many years and may need to be refreshedmay need to be refreshed
• New technology offers flexibility that can improve processes and exploit DR environment
– IT will implement new technologies – BC group needs to make sure new technology integrates with DR strategies
– Look for opportunities to provide value-added services
18
• DR requirements need to be revisited to make sure they are meeting business needs and risks– Evaluate better methods to develop, test and evaluate DR
recovery plans and capabilities
Strategies: Strategic Restructure of DR Planning
• Recovery Plans are owned and managed by the Technical and Business Areas – Business owners are responsible for readiness of DR plans
i l di i t lid ti d i ffincluding maintenance, validation and sign-off– New recovery systems funded by business owners
• Reorganize and classify DR plans according to Business Services and criticality– Adjust business continuity requirements and practices based
on the criticality and risk factors gained from reclassification• Redesign the DR planning processRedesign the DR planning process
– Implement a new processes to maintain and test recovery systems and plans independent of formal DR exercises
– Modify DR exercises to focus on mission-critical Business Services
19
Strategies: Risks associated with DR Planning Cycle
Release Cycle/ Change Process2. Recovery System
Risk Exposure
DR Exercise Cycle
Development Testing Prod. Implementation
DR
Exercise
Y/NY
Risks:1. Production systems are updated on a regular basis
– Recovery systems updated and tested less frequently as a norm– Recovery systems can remain out-of-date until participation in an
2-4+ weeksN
Plan Update Exercise Publication1. Recovery System
Risk Exposure
y y p pexercise
2. Recovery systems and environment are maintained in conjunction with a planned DR exercise
– Systems remain out-of-date until the exercise planning cycle
20
Strategies: Classify Systems & Functions to Business Services
• Type: Core, Products, Corporate, Infrastructure• Business Services: Assign systems and business functions
to Business Servicesto Business Services• Criticality: Assign business value (Mission Critical, Critical,
Vital) to Business Service to drive recovery requirements
Business Function
Business Function
Business Function
Business Function
DR System DR System DR System
Mission Critical
Business Service
21
Business Function
Business Function
DR System DR System VitalBusiness
Service
Strategies: Business Services Approach
• DR Plans are organized according to:– Business Services: Assign systems and business functions to Business
Services
– Criticality: Assign business value (Mission Critical, Critical, Vital) to Business Services
• Business continuity requirements and practices are based on the criticality and risk factors associated with Business Services
• Business continuity planning efforts are focused on ensuring the resiliency of mission-critical Business Services
22
Business Service
Mission Critical
Business Function
Business Function
Business Function
Business Function
DR System DR System DR System
Strategies: Redesign DR Planning Process
System Release
Cycle Requirements Design Development Testing Production
RecoveryRecovery System
Update Process Analysis Plan Update System Recovery
Test
• Recovery System Analysis Meetings• Recovery Plan Updates
• Procedure Validation• Owner sign-off on recovery status
Recovery System Update Process
Modify DR Exercise Program
23
Modify DR Exercise Program• Modify exercise approach to focus on Core Business Services
• Conduct Ad-hoc DR Exercises (limit size and scope)
• Test DR Plans for Deferred Systems
Strategies: DR Planning Cycle and Exercise Process
Procedure Validation
Test Cycles
Business and Technical. Owner
Sign-offYear-endSystem
Year-End Business Service Evaluation
Essential
System Changes
Recovery Plan
Update
Team LeaderSign-off
(Quarterly)
ProcedureValidation
YesBusiness
Service ExercisesParticipate
DRExercise
R t f h t h
No
yEvaluation
Year-endEvaluation
by BusinessService
Recovery Strategy
D t
Business Service
Evaluation
24
Essential Systems:
Repeats for each system change Document Evaluation
Controls:1. Recovery Plan Updates/Sign-offs2. Procedure Validations/Sign-offs3. DR Exercise Participation4. Year-end System Evaluation
Final Thoughts
• New technology will have a significant impact on DR strategies
• Hardware and software cost savings should be realized• Hardware and software cost savings should be realized• Enhanced resiliency services will reduce risks and provide
significant value to the company• A transition period will be required to allow tools/processes
and business applications to catch-up• Existing DR plans and strategies will remain for less critical
systems• Re-engineer DR strategies to address changing business
needs and technology advancements• Integrate business operations with technology to ensure
business service resiliency.
25
