Lessons Learned From the Texas Gulf Coast

ACUTA 2009Palm Springs, California

About Your SpeakersTexas State University System

Dr. Michael E. Dobe, Sr., Associate Vice President, Chief Information Officer & Adjunct Professor at Lamar University (Beaumont, Texas)

http://www.linkedin.com/in/mdobe Texas A & M System

Mr. Steven Conway, Director of Computing and Info Services at Texas A&M University at Galveston (Galveston, Texas)

http://www.linkedin.com/pub/5/b7/24a

About the Checklist

Steven Conway

Texas A&M University at Galveston (TAMUG)

ACUTA/ATIS Hurricane Preparedness Project

Formed Committee May 2007Reviewed and Modified ATIS documentPublished ACUTA Hurricane Checklist Oct

2007

http://www.acuta.org/?2026

Hurricane Preparedness

Checklist

ACUTA - The Association for Information Communications Technology Professionals in Higher Education

Developed in cooperation withATIS – The Alliance for Telecommunications Industry Solutions

Hurricane Katrina, August 2005, Terra MODIS True Color Image

Courtesy of Louisiana State University

Committee Amy Phillips, Louisiana State University Health Sciences Center Frank O'Quinn, Louisiana State University Sandra Thompson, University of West Florida James Shea, Boston University Jeanne Spinosa, Johnson & Wales University Garret Yoshimi, University of Hawaii George Denbow, University of Texas at Austin Michele Narcavage, University of Pennsylvania Steve Ellis, University of Notre Dame Steve Conway, Texas A&M University at Galveston Walt Magnussen, Texas A&M University - Task Force Chair Veronica Lancaster, ATIS Brad Ruark, ATIS /AT&T Jeri Semer, ACUTA Executive Director Aaron Fuehrer, ACUTA Staff Liaison

ACUTA Hurricane Preparedness Checklist

Using the Checklist As An After-Action Review Tool

Perspectives From Beaumont

Dr. Michael Dobe, Sr.Lamar University

About Lamar UniversityLocated in Beaumont, TX (Approx 40 Minutes

from Louisiana Border)Central Data Center in Beaumont Serves

Three Campuses:Lamar UniversityLamar Institute of TechnologyLamar State College- Orange

Total Students Served – Approximately 16,000

Proximity of Beaumont and Galveston

119 mi – about 1 hour 57 mins

Rita, Edouard, Gustav, IkeHurricane Activity in SE Texas

Rita – September 2005 (CAT 4 With Major Damage to Beaumont)

2006 and 2007 Atlantic Hurricane Seasons Quiet in SE Texas

Edouard – August 2008 (Missed SE Texas)Gustav – September 1, 2008 (Minor Impact in

SE Texas)Ike Hit SE Texas Less Than Two Weeks After

Gustav…

Hurricane Ike’s Path

http://en.wikipedia.org/wiki/File:Ike_2008_track.png

Hurricane Ike Timeline Lamar evacuated campus 10 Sept 2008Hurricane Ike landfall Evening of 12-13 Sept 2008Low Lying Areas of Beaumont Flooded, Extensive

Flooding in Orange County, Bridge City Hardest HitLamar University Campus in Beaumont Did Not FloodLibrary, Family and Consumer Sciences, and

Montaigne Center Buildings Had Roof Damage24 Sept 2008 Essential Staff Returned to Campus25 Sept 2008 Classes Resumed at Lamar University

Assessing Services During IkeLU Lessons LearnedStorm FactsContact ListsEmergency Sites and

LodgingAdministration,

Communications, and Manpower

Safety, Security and Training

InventoryPhysical AssetsPower and FuelNetwork ServiceMain Computer Center

and Key Telecommunications Hubs

Long-Term PreparationsPost-Storm ActivitiesDamage Assessment and

Post-Storm Activities

Storm Facts NOAA WebsiteMajor News Media: CNN, Weather Channel

and Storm Pulse (http://www.stormpulse.com/)

Importance of Local News Media: KFDM and Entergy

Government Contacts: Jefferson County and FEMA

Local Contacts: State Police and Beaumont Police

Contact Lists Updated and Distributed Internal Contacts

For GustavLeadership Team Phone ListPocket Card for Satellite Phone NumbersITS Contact List

External Contact Lists Maintained By Staff Sections

Emergency Sites and Lodging Emergency Control Site Remained on

Campus in BeaumontTemporary Lodging and Shelter Arranged by

IndividualsDisaster Recovery/Business Continuity Site at

San Marcos State University

Texas State University Systemhttp://www.tsus.edu/

TSUS Institutions (East Texas Locations in Bold)Lamar University Sam Houston State University Sul Ross State University Sul Ross State University Rio Grande

College Texas State University-San Marcos Lamar Institute of Technology Lamar State College-Orange Lamar State College-Port Arthur

Sam Houston State in Hunstville

153 mi – about 2 hours 29 mins

TOO CLOSE FOR COMFORT

Stephen F. Austin State

128 mi – about 2 hours 36 mins

NOT MEMBER OF TSUS

San Marcos State University

251 mi – about 4 hours 15 mins

During Ike – Approx. 16 hrs

LAMAR UNIVERSITY BC Site

Administration, Communications and Manpower Importance of Clear Chain of Command

and Clear Delineation of ResponsibilitiesITS Set Up Two Teams: Offsite B/C Team

and Onsite Command Center TeamCommunication with Employees via LU

Website and ConnectED (Now Blackboard Connect) to E-Mail and Phones.

Communications Methods: Cell Phones, Satellite Phones, Land Lines and SIP Phones for Key Employees

ACUTA NOTE: Effective Leadership Was Key to Success

ACUTA Note: Importance of Coordination of Operations Functions

ACUTA NOTE: Importance of Coordination of the Academic Response

Administration, Communications and Manpower ITS Hunkered Down With LUPD in Beaumont

Daily Meetings for Coordination in Beaumont on Campus

Reporting and Notification Lead By President and Leadership Team

Conference Bridges For Coordination With Remote Staff Managed by Network Services

Public Coordination Handled by LU PDManaged Through Office of VP for Finance and Amin

Supplemental PersonnelAccountability and Time ReportingCosts

LU IT Services Staff DispositionLocal Core Team Stayed on Campus

ThroughoutRemote Core Team Relocated to San Marcos

Area – Worked on Ordering Replacement Systems

Remaining Staff Dispersed With FamiliesAT&T Conference Bridge to Maintain Contact

With Dispersed Staff

Onsite IT Service Lead: John Genuardi, Customer Support Director

Onsite Network Support: Patrick Stewart, Manager of Network Services

Onsite Data Center Lead: Srinivas “Sri” Vandaraj, Senior Security Analyst

Safety, Security and Training President’s Weekly Security Meeting

Initiated as Outcome of RitaSystem Access Managed By Onsite Core ITS

Team (Issues With Passwords: Need One-Time Password System)

Physical Re-Entry Credentials Provided by Chief of Police

ITS and PD Coordinated Security Cameras Shutdown and Start-upTraining for ConnectedED

Inventory and Physical AssetsImpact of Key Staff ResignationPlanning: AVP (IT Infrastructure and Security)

Responsible for Availability of Emergency Supplies and Work Supplies

AVP Had Resigned Between Gustav and Ike!Single Point of Staff Failure – Transitioned

Responsibilities Quickly …Equipment Inventory, To Include Network and

Data Center Backup and Recovery Plans Needed to Be Transitioned ON THE FLY! (New Role of Onsite Lead)

Note on Vehicle Planning: Golf Carts Proved Critical After Storm

Power and Fuel: Handled By Facilities ManagementStaff Continuity Maintained - Gustav to Ike Communication with Local Fuel VendorsCommunication and Coordination With Power

CompaniesEngine/Genset (portable and permanent)Fuel TanksBatteriesRectifiersStaging and Deployment

Network ServicePreventative Measures:

No Redundant Routes for Network (Big Exposure) LEARN Connectivity Initiative Q 1 of 2009Half-Completed VoIP Migration (Half Cisco/Half

Avaya)Priority Areas:

Network Services Spread Between Windows Services Group (DHCP) and Unix Group (DNS).

Failure to Standardize Led to Confusion and LAN Service Outages During the Storm (Premature AD Shutdown).

Failure to Document All Services Creates Risk

Significant Telecomm EventsGOT LUCKY -- LU OC-3 Circuit to Houston Stayed Up

Throughout Event – NO INTERNET CONNECTIVITY OUTAGE (Gigaman to Orange Also Stayed Up)

Onsite Team Kept Campus Services Running Until Night of Landfall (Sept. 12-13)

Throughout Storm VoIP access via WAN using SIP Phones (Ordered More Broadband Cards After Storm)

Cell Service Unreliable Beaumont to Remote SitesResidential lines out for over several weeksSatellite Phones Proved Difficult to Use

Main Computer Center and Key Telecommunications Hubs Success Story: No Systems Shutdown

Sequence in Place For Data Centers. Planning Initiated in Gustav Continued Into Ike Lead Up

Downside to Not Running Your Own Power:Power Failure Exposure Due to Unmet Power

Requirements in Data CenterConcerns re: Lack of Fuel Led Local Team to Bring

Down Systems After EventTape Backups: Contract for Off Site Had Just

Lapsed! Can You Sing “Do the Hustle”?Wind and High Water: Failure to Stock Pile

Sandbags

Long Term Preparations Maintain Year-Round

Example of Successful Planning: Replication of Website and Content Management Service

Criticality of ERP Disaster Recovery/BC Site at San Marcos

Hardware In Place in San Marcos Software to Be Configured With Data Replication (Tested)

By Hurricane Season 2009Long-Term Plans/Prevention

Move to the Cloud: Portal and Mail Outsourcing Initiatives Importance of Service Continuity: Redesign of Network

Services (Standardization Under Network Services Group)

Post-Storm Activities Led By Onsite Team Lead Employee SafetyContactsConference BridgesCostsTemporary Lodging/ShelterExtra Work ForceSafetyEmergency Supply AvailabilityPower and Fuel

Flooding in Low-Lying Areas

Damage to People’s Homes

Damage to Local Businesses

Damage Assessment and Service Restoration Announcement Via ConnectED on Return of

Staff to Campus (September 24th), Return of Students to Campus (September 25th)

Class Schedule Readjusted by Office of the Provost, With Resumption of Classes on September 25th

President’s Faculty/Staff Convocation on September 25th

Perspectives From Galveston

Steven Conway

Texas A&M University at Galveston (TAMUG)

Hurricane Ike TAMUG evacuated campus 10 Sept 2008Hurricane Ike landfall 12 Sept 200875% of Galveston was floodedTAMUG did not flood, 2 buildings had roof

damageTAMUG restarted classes at TAMU in College

Station TX Wed 24 Sept91% of the students returned20 Jan 2008 Classes resume in Galveston

Significant Telecomm EventsCell service stayed up throughout storm-Cat 2UTMB Gigaman Circuit to Houston stayed upTAMUG-UTMB Gigaman available 24 Sept

when power returned on campusT1’s out for over a monthResidential lines out for over 6 weeksVoIP access to College Station via WAN

Hurricane Ike Flooding Galveston

A house burns uncontrolled in a flooded neighborhood as Hurricane Ike approaches the Texas coast, Friday, Sept. 12, 2008, in Galveston, Texas. (AP Photo/Matt Slocum)

TAMUG Boat on Causeway