1
TASSCC Annual Conference 2010
Business Resiliency Business Resiliency
PlanningPlanning-Business Continuity Management--Business Continuity Management-
William Tompkins, CISSP, CBCP
Teacher Retirement System of Texas
August 2, 2010
2
William Tompkins is Business Continuity/Disaster Recovery Coordinator and William Tompkins is Business Continuity/Disaster Recovery Coordinator and Information Security Officer at Teacher Retirement System of Texas. He Information Security Officer at Teacher Retirement System of Texas. He has more than 26 years of technical, managerial and consulting experience has more than 26 years of technical, managerial and consulting experience in information technology and more than 18 years in business continuity in information technology and more than 18 years in business continuity and information security planning. He is a Certified Business Continuity and information security planning. He is a Certified Business Continuity Professional and a Certified Information Systems Security Professional. Professional and a Certified Information Systems Security Professional.
He is the current President of the Association of Contingency Planners chapter He is the current President of the Association of Contingency Planners chapter in Austin.in Austin.
William was elected to the ISSA Hall of Fame in 2006 by the ISSA William was elected to the ISSA Hall of Fame in 2006 by the ISSA International Board of Directors. (International Board of Directors. (IInformation nformation SSystems ystems SSecurity ecurity AAssociationssociation))
Mr. Tompkins holds two Bachelor of Science degrees, Psychology and Mr. Tompkins holds two Bachelor of Science degrees, Psychology and Computer Information Science, from Troy State University in Alabama and Computer Information Science, from Troy State University in Alabama and Certification in Risk Management from University of Texas at Austin Certification in Risk Management from University of Texas at Austin Division of Continuing Education.Division of Continuing Education.
William TompkinsWilliam Tompkins
3
In this session we’ll overview In this session we’ll overview business resiliency practices business resiliency practices at at Teacher Retirement System of Texas, including , including our planning & maintenance our planning & maintenance practices, coordination with practices, coordination with other agencies, business other agencies, business partners, and our contracted partners, and our contracted recovery service provider.recovery service provider.
4
AgendaAgenda
Why ?Why ?
How ?How ?
What ?What ?
Q & AQ & A
5
6
Are we ready?
PresumptionsPresumptions
RealityRealityRealityReality
*from Managing Managers: A Case Study by Philip Jan Rothstein; Copyright 1995, Rothstein Associates Inc.
Versus
7
PresumptionPresumption
RealityReality
The “wizards” (in IT Div.) could handle any crisis and business would be operational within a few hours.
At best, recovery from a MAJOR disruption could take 30-36 hours.
*from Managing Managers: A Case Study by Philip Jan Rothstein; Copyright 1995, Rothstein Associates Inc.
Are we ready?Are we ready?
8
RealityReality
IT Div. “automatically” integrates all diverse technology and platforms into the Disaster Recovery Program.
IT Div. did not implement -OR- operate many of these platforms.
Are we ready?Are we ready?
PresumptionPresumption
*from Managing Managers: A Case Study by Philip Jan Rothstein; Copyright 1995, Rothstein Associates Inc.
9
RealityReality
No matter what the cause or scope of disruption ... IT Div. would recover all data accurately AND to the point of failure.
At best, recovery would be to the prior night's backup and, most probably, to a point at least 3 to 4 nights prior.
*from Managing Managers: A Case Study by Philip Jan Rothstein; Copyright 1995, Rothstein Associates Inc.
Are we ready?Are we ready?
PresumptionPresumption
10
RealityReality
Data entry sections have manually filed source documents, so the data entered since the last backup is clearly identified.
?
?
PresumptionPresumption
Are we ready?Are we ready?
11
RealityReality
Well-respected practitioner had a very good program.
Senior management was dissatisfied with the program…because the organization’s professionals were not familiar with the real business processes.
Are we ready?Are we ready?
PresumptionPresumption
12
TimelineTimelineTimelineTimelineReturn to Normal
Operations
Evaluate&
Decision
Restore(most data &
some infrastructure)
Recovery(weeks to months?)
Begin ResponseMobilize people &
Notify recoverycontractors
Min. 4 hrs.Max. 12 hrs.
12 – 24hours
48 hours Staff begins re-entering Tues work no earlier than Saturday morning
72 hrs72 hrs
13
By the end of this session . . . better understanding of business resiliency
Administrative activitiesAdministrative activities
Planning activitiesPlanning activities
Technical activitiesTechnical activities
User educationUser education
14
Administrative activitiesAdministrative activities
Policy Policy (Business Continuity Management Policy)DefinitionsDefinitions
Business Impact Assessment (BIA)Business Impact Assessment (BIA)Mission criticalMission critical
RolesRolesBusiness Continuity/Disaster Recovery Business Continuity/Disaster Recovery
CoordinatorCoordinator -vs- Business Continuity Planner -vs- Business Continuity Planner
15
RolesManagement support:Management support:
Executive management Executive management Project initiation, scope, final approval, ongoing supportProject initiation, scope, final approval, ongoing support
Senior business unit management Senior business unit management Identifies and prioritizes time-critical systemsIdentifies and prioritizes time-critical systems
Functional business units Functional business units (departments)
Participate in implementing and testingParticipate in implementing and testing
16
Administrative activitiesAdministrative activities
Reporting Reporting Annual BCP Annual BCP – a summary report – a summary report
includes copy of up-to-date BIA and dates includes copy of up-to-date BIA and dates of IMT Plan, Incident Response Plan, of IMT Plan, Incident Response Plan, business unit continuity plans business unit continuity plans & IT’s DR & Telecommunications plans& IT’s DR & Telecommunications plans
After-action of Hot Site ExerciseAfter-action of Hot Site ExerciseResults of “primary” & “secondary” Results of “primary” & “secondary”
objectivesobjectivesAnnual Risk Assessment
17
Program Goal( from “Policy” )
……to prepare to counteract interruptions to prepare to counteract interruptions
to TRS’ business activities and to to TRS’ business activities and to
protect critical business processes protect critical business processes
from the effects of disasters or major from the effects of disasters or major
failures of information systems and to failures of information systems and to
ensure their timely resumptionensure their timely resumption
18
PlanningPlanning
19
What is BIAWhat is BIA A Business Impact Analysis (‘BIA’) identifies and A Business Impact Analysis (‘BIA’) identifies and
prioritizes the critical business processes supported prioritizes the critical business processes supported by the technology infrastructure. by the technology infrastructure.
BIA Key Components:BIA Key Components:
Identifies the impact of potential resource lossIdentifies the impact of potential resource loss
Identifies the minimum resources needed to recoverIdentifies the minimum resources needed to recover
Prioritizes the recovery of processes and supporting Prioritizes the recovery of processes and supporting systemssystems
Establishes the escalation of that loss over timeEstablishes the escalation of that loss over time
20
Impact priority considerationsImpact priority considerations{not in priority sequence}
★Required by law
★Critical or essential business need
★Inaction (or incorrect action) violates fiduciary duty
★Inaction causes harm
★Impacts large number of people
★Severe adverse impact on TRS’ mission, functions, or reputation
21
BIA QuestionsBIA QuestionsWhat are the critical functions?
Why are they critical?How quickly does it need to be recovered? Why?Does it need to be recovered in the event of a disruption/disaster?
If it is not recovered as quickly as it needs to be, what will happen? So what? Who else would be affected?
22
Chart legend for following pages
25
Contingency PlanningContingency Planning
Risk Management identifies risks that Risk Management identifies risks that require contingency plansrequire contingency plans
Risk decisions are based on BIA detailsRisk decisions are based on BIA details
Contingency plans - business decisions Contingency plans - business decisions based on real numbers and facts.based on real numbers and facts.
26
Contingency Plans Are:Contingency Plans Are:
Interim recovery measuresInterim recovery measures that ensure survival of the that ensure survival of the organization during a disaster event by providing for organization during a disaster event by providing for continuity of its critical business functions.continuity of its critical business functions.
Long term outage provisions Long term outage provisions
Critical system relocation proceduresCritical system relocation procedures
Personnel issues Personnel issues –– get the right people to right place get the right people to right place
(Internal) Temporary business operation modes (Internal) Temporary business operation modes
(External) How to deal with customers, partners, and (External) How to deal with customers, partners, and shareholders through different channelsshareholders through different channels
27
Planning activitiesPlanning activities
Business Continuity Plan ?Business Continuity Plan ?
Not exactly . . .Not exactly . . .
Business Resiliency Program ?Business Resiliency Program ?
YesYes
28
PlansIncident Management Team PlanIncident Management Team Plan
Single reference for Exec & Sr. MgmtSingle reference for Exec & Sr. Mgmt
Crisis Management Plan*Crisis Management Plan*After initial ‘triage’ …helps clarify eventAfter initial ‘triage’ …helps clarify event
What happenedWhat happened
How seriousHow serious
What to do nextWhat to do next
Incident Response Plan*Incident Response Plan*Addresses initial stages of any eventAddresses initial stages of any event
*Note: in some organizations, crisis management & incident response is the same*Note: in some organizations, crisis management & incident response is the same
29
Disaster Recovery PlansDisaster Recovery Plans
Enable quickly resuming operations Enable quickly resuming operations for most critical unitsfor most critical units
Network infrastructureNetwork infrastructure
IBM Mainframe IBM Mainframe (business class enterprise server)
& a midrange system& a midrange system
Telecommunications Telecommunications
Plans
30
PlansBusiness Continuity PlansBusiness Continuity Plans
Covers all critical and major business units (17), Covers all critical and major business units (17), provides detail for staff involved in early recovery effortsprovides detail for staff involved in early recovery efforts
Business UnitsGeneral Accounting External Public Markets Insurance (TRS Care)
Benefit Accounting Private Markets Insurance (Active Care)
Benefit Processing Trade Management General Counsel
Benefit Counseling Internal Public Markets Mail & Supplies Center
Member Data Services Investments (Research & Risk) Printing & Bindery
31
Site Restoration PlanSite Restoration Plan
Plan for restoration efforts by facilities Plan for restoration efforts by facilities
Vital Records Retention / Recovery Vital Records Retention / Recovery PlanPlan
Change Management PlanChange Management Plan
Include plan updates when technology or Include plan updates when technology or business process changesbusiness process changes
Plans
32
Technical Technical ActivitiesActivities
33
Data Back Up Data Back Up Routine BackupsRoutine Backups
Retention Retention Daily – 14 daysDaily – 14 days
Weekly – 6 monthsWeekly – 6 months
Monthly – 2 monthsMonthly – 2 months
Archive Archive [EOY] [EOY] (annual: Member data, based on (annual: Member data, based on annuitant)annuitant)
Mainframe{incl. Imaging (Filenet)}
Network
Nightly: All PROD data (approx. 6 hours) Weekly: Friday night (approx. 18 hrs)
Weekly: also incl. Devl. & Test data (11 hours) Mon. – Thu.: incremental backups
Monthly: same as weekly Monthly : “last Friday” copy
34
Actual “Hot-site” exerciseActual “Hot-site” exercise
Investment Div. has been actively Investment Div. has been actively involved in at least 6 exercisesinvolved in at least 6 exercises
Emergency Call Tree Exercise Emergency Call Tree Exercise
Tabletop exerciseTabletop exercisea sit down desk-check with the team a sit down desk-check with the team
leaders and team membersleaders and team members
Testing & ValidationTesting & Validation
35
User User EducationEducation
36
User AwarenessUser Awareness
Providing Awareness, leads to…
Understanding
Change in AttitudeChange in Attitude
Change in Behavior!Change in Behavior!
37
Executives and Senior Managers/DirectorsExecutives and Senior Managers/Directors
ongoing familiarization sessions w / Sr. Mngt.ongoing familiarization sessions w / Sr. Mngt.
Business Unit Managers & Team LeadersBusiness Unit Managers & Team Leaders
considerations as business processes and business considerations as business processes and business
partners changepartners change
Regular Staff, Temp Hires & ContractorsRegular Staff, Temp Hires & Contractors
introductory classes given to new employeesintroductory classes given to new employees
User AwarenessUser Awareness
38
External External “Partners”
39
PrivatePrivate““Hot-site” contractHot-site” contract
Business Continuity & Resiliency ServicesBusiness Continuity & Resiliency Services
Off site storageOff site storage Tape backups & hardcopiesTape backups & hardcopies
Other State AgenciesOther State Agencies TCEQ TCEQ – Backup Command Center– Backup Command Center TPASSTPASS ((Tx Procurement & Support SvcsTx Procurement & Support Svcs) – Mail ) – Mail TxDOTTxDOT – Web site– Web site
External Contracts & MOUsExternal Contracts & MOUs
40
Self-assessmentSelf-assessment
41
www.theiia.org/technology
www.theiia.org/technology
42
Q U E S T I O N S ?Q U E S T I O N S ?Q U E S T I O N S ?Q U E S T I O N S ?
Thank YouThank YouThank YouThank YouWilliam A. TompkinsWilliam A. Tompkins
(512) 542-6787(512) 542-6787
[email protected]@trs.state.tx.us
William A. TompkinsWilliam A. Tompkins(512) 542-6787(512) 542-6787
[email protected]@trs.state.tx.us