Post on 04-Jun-2018
transcript
8/13/2019 Articulo CP Toigo
http://slidepdf.com/reader/full/articulo-cp-toigo 1/5
The disasters at the World Trade Center and Pentagon in mid-September,
while appalling in terms of their origin and horrific in terms of their human cost,
could ve been far worse, according to most emergency management experts. the hijacked aircraft had struck the WTC an hour later, as many as 50,000occu
pants and an incalculable number of tourists may have been counted among the
missing. Similarly, had the Pentagon not been undergoing significant remodeling
around the time that the building was attacked, the number of civilian and mili
tary personnel packed into those office spaces might have been much greater.
While these facts in no way mitigate the human tragedy that occurred, they
did have a significant impact on the overall death toll from the events. As smoke
and-rubble disasters go, the direct impact of the Sept. 11 terrorist attacks was ac
tually more limited than many. Within the past decade, for example, earthquakes
in Japan and Turkey claimed a larger number of lives in a much shorter amount
of time.
Aside from the social and political consequences of Sept. 11, perhaps the
most extraordinary thing about the disaster was that so few of the impacted com-
, panies appear 10 have had any sort of disaster recovery plan. Of the 440-odd
businesses occupying the WTC, and the numerous governmental entities in the
Pentagon, only a small subset-perhaps as few as 200-evidenced pre-planned
continuity strategies. My estimate is based on press accounts of the number offirms that formally declared a disaster and activated their contracts with any of
the several leading hot site vendors, such as Comdisco, IBMBusiness Continu
ity and Recovery Services, Sungard Recovery Services, and HP Business Continu
ity and Recovery Services. A hot site contract provides for a facility, computer
equipment and networks that can be put rapidly into service to replace a sub
scriber s production ITinfrastructure if normal operations are interrupted by a
disaster.
To be generous, a few companies may not have needed the services of a hot
site vendor in the wake of the disaster. In some cases, only branch office opera
tions were hosted within the WTCor the Pentagon, rather than a primary head
quarters or important data center. In a few other cases, companies may haveinstead used homegrown recovery strategies and capabilities that didn t require
the participation of a commercial service provider.
Even with these exceptions factored in, however, the number of companies
that had prepared for the possibility of a disaster were well in the minority. The
sad truth is that, as in the case of the 143 companies that simply disappeared in
Jon William Toigo
Originally published in nt er p ri se Sy st ems December 2001.Reprinted by Permission.
By
PREPARING FOR THE UNTHINKABLE: CONTROL THE DAMACE
Chapter Conclusion
8/13/2019 Articulo CP Toigo
http://slidepdf.com/reader/full/articulo-cp-toigo 2/5
Plan for Total Disaster
Assume the worst, that critical infrastructure components over which
businesses have no control (including telecommunications, power and trans
portation) are unavailable. This was certainly the casefor hundreds of busi
nesses located near the WTC. Most lost power, communications, and, in
many cases, physical access to their facilities due to police and emergency
management cordons. According to one person with officesnear the disaster
area, O ur facilitywasn t directly impacted by the attack, but we lost all tele
phone communications with our [financial] clients and all of our overseas
lines for almost a week following the event. We had to activate our disaster
plan, despite the fact that we had no visible damage. The company, and
many of its neighbors, were part of the secondary disaster that almost in
evitably follows any regional disaster: Their business operations were
stopped cold, despite the fact that they were not directly in the path of the
terrorist-controlled aircraft. A well..designed disaster recovery plan takes the
worst-case scenario as its premise and is designed for modular implementa
tion in response to any lesser disaster events that confront the business.
2. Focus on KeyAssets
Themost important assets of an organization cannot be replaced. Disaster
recovery strategies come in two basic flavors: replacement or redundancy.
Since it s impossible to replace skilled personnel or data, a redundancy
strategy should be implemented. In terms of data, this means implementing
a data mirroring or tape backup strategy and ensuring that it s scrupulously
observed and periodically tested.
Many spokespersons for companies in the World Trade Center reported
that they had plans place to account for everything-except for the loss of
key personnel that resulted from the attack. According to a spokesperson
for the Port Authority of New York and New Jersey, We didn t anticipatean instantaneous loss of somany senior managers. In the blink of an eye, the
fire department lost most of its upper echelon. I remember being asked by a
fireman whether I had seen his battalion commander, captain, lieutenant or
any other officer. I hadn t and I didn t realize until long afterward that we
had lost somany of the hierarchy when the towers collapsed.
the months and years following the 1993bombing of the WTC, many of the com
panies that endured the Sept. 11 tragedy without a continuity plan will not be
around this time next year. These companies will learn their lessons about the im
portance of disaster recovery planning the hard way, adding further pain and an
guish to the already sad memory of that awful event.
Based on published reports, along with interviews I conducted in early Oc
tober with several WTC survivors, some lessons can be gleaned that may helpplanners to safeguard mission-critical business processes from future disasters.
Most have to do with the settings and circumstances in which the disaster recov
ery capability may need to be activated and used.
hapter Conclusion
8/13/2019 Articulo CP Toigo
http://slidepdf.com/reader/full/articulo-cp-toigo 3/5
As awful as it is to contemplate such human loss, a redundancy strategy
may mean cross-training numerous personnel to perform mission-critical
tasks, then dispersing them around the corporate campus or into field of
fices. Companies most profoundly impacted by the Sept. 11 attacks will be
those who lost not just electronic- and paper-based information assets, but
irreplaceable personnel.
3. Size Doesn t MatterA lesson that many organizations are learning the hard way from this in
cident is that they ve seriously underestimated the importance of data
stored on PC hard disks. When DR planners and IT managers in most orga
nizations think about mission-critical data, they tend to focus on big iron.
That means large arrays, large network-attached storage volumes, storage
area networks or mainframe DASD.
However, the lowly PC, with its 30 to 100GBhard disk drive and innocu
ous Excel spreadsheet used to track key corporate financial hedges, may
have significantly greater importance from a business recovery standpoint
than all the data in the corporate ERP system. The company that survives a
disaster will be one that has ferreted out all of these small-but-critical apps
so that they can be replicated off-site.
One planner whose company is recovering from the attack reports that,
We are discovering that the really important data was on hard disks of PCs
and laptops that were never backed up and didn t survive the disaster. 4. Work with Law Enforcement
For the first full week following the disaster, police and emergency man
agers were not allowing personnel into the cordoned area, which extended
several blocks from the actual WTC disaster site. When authorities began
opening the area to some traffic, only those who were able to show that they
resided in the restricted areas were allowed to pass.
Survivor companies recognize that their continuity plans will need to ex
ecute under the aegis of law enforcement and public safety professionals
who are less interested in how they re going to access offices to power
down equipment or take last-minute backups than they are in preventing
looting and ensuring public health and safety. It s a good idea to meet with
local civil emergency management and police agencies and to obtain clear
ances for your corporate ID badges in advance of any disaster. But, don t
rely on the clearances counting for much in the event of a major calamity.
5. Expect People s Best-and Worst
The spirit is willing, but the flesh is weak. The stories of courage coming
out of the WTC and Pentagon disasters are more than hero building. It san almost signature characteristic of disasters that they tend to bring out the
best-and the worst-in people. Consider the story of the man who lost his
briefcase under a car where he sought shelter from the dust and debris of
the second tower collapse. Several days later, the briefcase was returned by
a rescue worker who had discovered it while removing the automobile from
Chapter Conclusion
8/13/2019 Articulo CP Toigo
http://slidepdf.com/reader/full/articulo-cp-toigo 4/5
the rubble. The man was delighted to see that the briefcase still contained
his wallet, which was filled with cash and other personal effects. However,
a day or two later, the man also discovered that his credit-card number was
being used to purchase goods and services all over the city, apparently by
the same man who had returned the case, and that this had been going on
almost from the day of the disaster. The point is that you cannot assume
unilateral heroism in a disaster. The purpose of rehearsing and testing a disaster recovery plan is not to teach recovery team members how to perform
procedures by rote, but to get them acclimated to thinking rationally in the
face of a great irrationality. Human nature also requires that security safe
guards be provided in the recovery environment.
6. Watch Out for Third Parties Affected
Be careful about planning assumptions that involve third parties. Some
companies continue to use a next-box-off-the-line approach in their strate
gies for replacing hardware in the wake of a disaster. That is fh ir plan is to
replace damaged components by requesting a priority shipment of new
gear from suppliers as soon as possible following the disaster. The shut
down of air transportation following the Sept. 11 calamity compromisedquick hardware replacement for many firms.
One survivor reported that his company was running on batteries in the
days following the attack. Power was sketchy and we wanted to mirror
our NAS [network-attached storage] across the Hudson River. Our vendor
told us that because of a change in their sales territories, we couldn t get
more product from the local reseller who had originally sold us the gear. t
had to be delivered to us by our new direct sales account representative, lo
cated on the West Coast. Our reseller was livid-he was close by with gear
on his shelf that he could get to us n an hour, but he was being told he
would violate his reseller agreement if he supported us. Ultimately, he told
them to go to hell and brought us what we needed.
For reasons like this, it s a good idea to maintain critical spare compo
nents at a secure off-site facility.
7. Plan Employee Work Space .
Many disaster recovery plans stop at provisioning for systems and network
replacement. They don t provide for new user work locations. This not only
compromises the recovery timetable, it can lead to employee confusion. Going
forward, companies may want to consider using application service providers
or managed Web-hosting providers to make mission-critical applications ac
cessible via dial-up connections or the World Wide Web. Such an approach
would enable work-at-home strategies for workers who are equipped to doso-a useful hedge until suitable replacement work areas can be located.
8. Try to Avoid the Media
You ll want to establish a command center away from the media. After
the bombing of the WTC in 1993, one Big Five accounting firm established
an ad hoc command center in a nearby office complex. The location proved
hapter Conclusion
8/13/2019 Articulo CP Toigo
http://slidepdf.com/reader/full/articulo-cp-toigo 5/5
to be a preferred backdrop for television journalists covering the event. In
advertently, TV cameras captured images of a whiteboard containing two
credit-card account numbers intended for use by the company s recovery
teams as they secured supplies and equipment. The numbers were broad
cast around the country and were repeatedly misused by nefarious viewers
of the program. After Sept. 11,no such mistakes were made. The need to es
tablish a command center away from the disaster site, and to deal with
media through the vehicle of corporate communications or experienced PRfirms, seemed to be recognized by most disaster-stricken firms.
9. Consider Your Workers
The impact of any disaster, whether the result of terrorism or some natural
calamity, exacts a toll on the psyches ofworkers. Shorter workdays, half shifts,
on-site counseling and other compassionate considerations may aid more in a
successful recovery than all the logistics and plans combined. The good news
is that most companies, including many impacted by the wr incident, find
that concerns about employee availability in the aftermath of a disaster are
often unfounded. In most cases, disasters have a galvanizing impact on company teams: More than one planner reported that he needed to turn away per
sonnel offering to assist recovery efforts. It s important to keep employees
apprised of the situation, rather than having them draw conclusions from TV
reports. When you need employees help, most planners report, it s available.
10. Reward Innovation
Disaster recovery plans are not scripts for recovery efforts; they re guide
lines at best. Given the shifting battlefield of recovery efforts, preplanned
approaches sometimes need to give way to expediency.
According to one company spokesperson, We had planned to keep in
touch with our recovery teams by cell phone. But there were times after the in
cident when the cells in Manhattan were completely saturated. We startedusing runners to carry verbal messages from one site to another. [Sticking to
the cell phone plan] would have slowed down our recovery a lot, but some
teams simply abandoned the planned procedures and moved forward on their
own initiative. As a result, we were able to keep the recovery effort on track.
Reports like this emphasize the importance of encouraging innovation
and creativity on the part of recovery teams. When team leaders feel em
powered to take the initiative, planners should reward this, either immedi
ately or in debriefing meetings well after the fact. fmistakes are made,
forgive them, at least in the short term, since customers, shareholders and
others who are waiting for a return to normalcy will do the same. Most will
understand that disaster recovery is difficult work.
Chapter 2 Conclusion