8/13/2019 Articulo CP Toigo

http://slidepdf.com/reader/full/articulo-cp-toigo 1/5

The disasters  at the World Trade Center and Pentagon in mid-September,

while appalling in terms of their origin and horrific in terms of their human cost,

could ve been far worse, according to most emergency management experts.  the hijacked aircraft had struck the WTC an hour later, as many as 50,000occu

pants and an incalculable number of tourists may have been counted among the

missing. Similarly, had the Pentagon not been undergoing significant remodeling

around the time that the building was attacked, the number of civilian and mili

tary personnel packed into those office spaces might have been much greater.

While these facts in no way mitigate the human tragedy that occurred, they

did have a significant impact on the overall death toll from the events. As smoke

and-rubble disasters go, the direct impact of the Sept. 11 terrorist attacks was ac

tually more limited than many. Within the past decade, for example, earthquakes

in Japan and Turkey claimed a larger number of lives in a much shorter amount

of time.

Aside from the social and political consequences of Sept. 11, perhaps the

most extraordinary thing about the disaster was that so few of the impacted com-

, panies appear 10 have had any sort of disaster recovery plan. Of the 440-odd

businesses occupying the WTC, and the numerous governmental entities in the

Pentagon, only a small subset-perhaps as few as 200-evidenced pre-planned

continuity strategies. My estimate is based on press accounts of the number offirms that formally declared a disaster and activated their contracts with any of

the several leading hot site vendors, such as Comdisco, IBMBusiness Continu

ity and Recovery Services, Sungard Recovery Services, and HP Business Continu

ity and Recovery Services. A hot site contract provides for a facility, computer

equipment and networks that can be put rapidly into service to replace a sub

scriber s production ITinfrastructure if normal operations are interrupted by a

disaster.

To be generous, a few companies may not have needed the services of a hot

site vendor in the wake of the disaster. In some cases, only branch office opera

tions were hosted within the WTCor the Pentagon, rather than a primary head

quarters or important data center. In a few other cases, companies may haveinstead used homegrown recovery strategies and capabilities that didn t require

the participation of a commercial service provider.

Even with these exceptions factored in, however, the number of companies

that had prepared for the possibility of a disaster were well in the minority. The

sad truth is that, as in the case of the 143 companies that simply disappeared in

Jon William Toigo

 Originally published in  nt er p ri se Sy st ems December 2001.Reprinted by Permission. 

By

PREPARING FOR THE UNTHINKABLE: CONTROL THE DAMACE

Chapter Conclusion 

8/13/2019 Articulo CP Toigo

http://slidepdf.com/reader/full/articulo-cp-toigo 2/5

  Plan for Total Disaster

Assume the worst, that critical infrastructure components over which

businesses have no control (including telecommunications, power and trans

portation) are unavailable. This was certainly the casefor hundreds of busi

nesses located near the WTC. Most lost power, communications, and, in

many cases, physical access to their facilities due to police and emergency

management cordons. According to one person with officesnear the disaster

area,  O ur facilitywasn t directly impacted by the attack, but we lost all tele

phone communications with our [financial] clients and all of our overseas

lines for almost a week following the event. We had to activate our disaster

plan, despite the fact that we had no visible damage. The company, and

many of its neighbors, were part of the secondary disaster that almost in

evitably follows any regional disaster: Their business operations were

stopped cold, despite the fact that they were not directly in the path of the

terrorist-controlled aircraft. A well..designed disaster recovery plan takes the

worst-case scenario as its premise and is designed for modular implementa

tion in response to any lesser disaster events that confront the business.

2. Focus on KeyAssets

Themost important assets of an organization cannot be replaced. Disaster

recovery strategies come in two basic flavors: replacement or redundancy.

Since it s impossible to replace skilled personnel or data, a redundancy

strategy should be implemented. In terms of data, this means implementing

a data mirroring or tape backup strategy and ensuring that it s scrupulously

observed and periodically tested.

Many spokespersons for companies in the World Trade Center reported

that they had plans   place to account for everything-except for the loss of

key personnel that resulted from the attack. According to a spokesperson

for the Port Authority of New York and New Jersey, We didn t anticipatean instantaneous loss of somany senior managers. In the blink of an eye, the

fire department lost most of its upper echelon. I remember being asked by a

fireman whether I had seen his battalion commander, captain, lieutenant or

any other officer. I hadn t and I didn t realize until long afterward that we

had lost somany of the hierarchy when the towers collapsed.

the months and years following the 1993bombing of the WTC, many of the com

panies that endured the Sept. 11 tragedy without a continuity plan will not be

around this time next year. These companies will learn their lessons about the im

portance of disaster recovery planning the hard way, adding further pain and an

guish to the already sad memory of that awful event.

Based on published reports, along with interviews I conducted in early Oc

tober with several WTC survivors, some lessons can be gleaned that may helpplanners to safeguard mission-critical business processes from future disasters.

Most have to do with the settings and circumstances in which the disaster recov

ery capability may need to be activated and used.

 hapter Conclusion

8/13/2019 Articulo CP Toigo

http://slidepdf.com/reader/full/articulo-cp-toigo 3/5

As awful as it is to contemplate such human loss, a redundancy strategy

may mean cross-training numerous personnel to perform mission-critical

tasks, then dispersing them around the corporate campus or into field of

fices. Companies most profoundly impacted by the Sept. 11 attacks will be

those who lost not just electronic- and paper-based information assets, but

irreplaceable personnel.

3. Size Doesn t MatterA lesson that many organizations are learning the hard way from this in

cident is that they ve seriously underestimated the importance of data

stored on PC hard disks. When DR planners and IT managers in most orga

nizations think about mission-critical data, they tend to focus on big iron.

That means large arrays, large network-attached storage volumes, storage

area networks or mainframe DASD.

However, the lowly PC, with its 30 to 100GBhard disk drive and innocu

ous Excel spreadsheet used to track key corporate financial hedges, may

have significantly greater importance from a business recovery standpoint

than all the data in the corporate ERP system. The company that survives a

disaster will be one that has ferreted out all of these small-but-critical apps

so that they can be replicated off-site.

One planner whose company is recovering from the attack reports that,

  We are discovering that the really important data was on hard disks of PCs

and laptops that were never backed up and didn t survive the disaster. 4. Work with Law Enforcement

For the first full week following the disaster, police and emergency man

agers were not allowing personnel into the cordoned area, which extended

several blocks from the actual WTC disaster site. When authorities began

opening the area to some traffic, only those who were able to show that they

resided in the restricted areas were allowed to pass.

Survivor companies recognize that their continuity plans will need to ex

ecute under the aegis of law enforcement and public safety professionals

who are less interested in how they re going to access offices to power

down equipment or take last-minute backups than they are in preventing

looting and ensuring public health and safety. It s a good idea to meet with

local civil emergency management and police agencies and to obtain clear

ances for your corporate ID badges in advance of any disaster. But, don t

rely on the clearances counting for much in the event of a major calamity.

5. Expect People s Best-and Worst

The spirit is willing, but the flesh is weak. The stories of courage coming

out of the WTC and Pentagon disasters are more than hero building.  It san almost signature characteristic of disasters that they tend to bring out the

best-and the worst-in people. Consider the story of the man who lost his

briefcase under a car where he sought shelter from the dust and debris of

the second tower collapse. Several days later, the briefcase was returned by

a rescue worker who had discovered it while removing the automobile from

Chapter Conclusion 

8/13/2019 Articulo CP Toigo

http://slidepdf.com/reader/full/articulo-cp-toigo 4/5

the rubble. The man was delighted to see that the briefcase still contained

his wallet, which was filled with cash and other personal effects. However,

a day or two later, the man also discovered that his credit-card number was

being used to purchase goods and services all over the city, apparently by

the same man who had returned the case, and that this had been going on

almost from the day of the disaster. The point is that you cannot assume

unilateral heroism in a disaster. The purpose of rehearsing and testing a disaster recovery plan is not to teach recovery team members how to perform

procedures by rote, but to get them acclimated to thinking rationally in the

face of a great irrationality. Human nature also requires that security safe

guards be provided in the recovery environment.

6. Watch Out for Third Parties Affected

Be careful about planning assumptions that involve third parties. Some

companies continue to use a next-box-off-the-line approach in their strate

gies for replacing hardware in the wake of a disaster. That is fh ir plan is to

replace damaged components by requesting a priority shipment of new

gear from suppliers as soon as possible following the disaster. The shut

down of air transportation following the Sept. 11 calamity compromisedquick hardware replacement for many firms.

One survivor reported that his company was running on batteries in the

days following the attack. Power was sketchy and we wanted to mirror

our NAS [network-attached storage] across the Hudson River. Our vendor

told us that because of a change in their sales territories, we couldn t get

more product from the local reseller who had originally sold us the gear.  t

had to be delivered to us by our new direct sales account representative, lo

cated on the West Coast. Our reseller was livid-he was close by with gear

on his shelf that he could get to us  n an hour, but he was being told he

would violate his reseller agreement if he supported us. Ultimately, he told

them to go to hell and brought us what we needed.

For reasons like this, it s a good idea to maintain critical spare compo

nents at a secure off-site facility.

7. Plan Employee Work Space .

Many disaster recovery plans stop at provisioning for systems and network

replacement. They don t provide for new user work locations. This not only

compromises the recovery timetable, it can lead to employee confusion. Going

forward, companies may want to consider using application service providers

or managed Web-hosting providers to make mission-critical applications ac

cessible via dial-up connections or the World Wide Web. Such an approach

would enable work-at-home strategies for workers who are equipped to doso-a useful hedge until suitable replacement work areas can be located.

8. Try to Avoid the Media

You ll want to establish a command center away from the media. After

the bombing of the WTC in 1993, one Big Five  accounting firm established

an ad hoc command center in a nearby office complex. The location proved

 hapter   Conclusion

8/13/2019 Articulo CP Toigo

http://slidepdf.com/reader/full/articulo-cp-toigo 5/5

to be a preferred backdrop for television journalists covering the event. In

advertently, TV cameras captured images of a whiteboard containing two

credit-card account numbers intended for use by the company s recovery

teams as they secured supplies and equipment. The numbers were broad

cast around the country and were repeatedly misused by nefarious viewers

of the program. After Sept. 11,no such mistakes were made. The need to es

tablish a command center away from the disaster site, and to deal with

media through the vehicle of corporate communications or experienced PRfirms, seemed to be recognized by most disaster-stricken firms.

9. Consider Your Workers

The impact of any disaster, whether the result of terrorism or some natural

calamity, exacts a toll on the psyches ofworkers. Shorter workdays, half shifts,

on-site counseling and other compassionate considerations may aid more in a

successful recovery than all the logistics and plans combined. The good news

is that most companies, including many impacted by the wr incident, find

that concerns about employee availability in the aftermath  of a disaster are

often unfounded. In most cases, disasters have a galvanizing impact on company teams: More than one planner reported that he needed to turn away per

sonnel offering to assist recovery efforts. It s important to keep employees

apprised of the situation, rather than having them draw conclusions from TV

reports. When you need employees help, most planners report, it s available.

10. Reward Innovation

Disaster recovery plans are not scripts for recovery efforts; they re guide

lines at best. Given the shifting battlefield of recovery efforts, preplanned

approaches sometimes need to give way to expediency.

According to one company spokesperson, We had planned to keep in

touch with our recovery teams by cell phone. But there were times after the in

cident when the cells in Manhattan were completely saturated. We startedusing runners to carry verbal messages from one site to another. [Sticking to

the cell phone plan] would have slowed down our recovery a lot, but some

teams simply abandoned the planned procedures and moved forward on their

own initiative. As a result, we were able to keep the recovery effort on track.

Reports like this emphasize the importance of encouraging innovation

and creativity on the part of recovery teams. When team leaders feel em

powered to take the initiative, planners should reward this, either immedi

ately or in debriefing meetings well after the fact.  fmistakes are made,

forgive them, at least in the short term, since customers, shareholders and

others who are waiting for a return to normalcy will do the same. Most will

understand that disaster recovery is difficult work.

Chapter 2 Conclusion