Objectives and performance metrics in oil spill response: The Bouchard-120 and Chalk Point spill responses A technical report submitted to the Coastal Response Research Center by Dr. Seth Tuler, Rebecca Kay, and Dr. Thomas P. Seager -and- 17 April, 2006 NOAA Grant Number: NA04NOS4190063. Project Number: 05-983. Social and Environmental Research Institute, Inc. 278 Main Street, Room 404, Greenfield, MA 01301 413.773.9955 www.seri-us.org Dr. Igor Linkov 58 Charles St., Cambridge MA 02141
Transcript
Objectives and performance metrics inoil spill response: The Bouchard-120
and Chalk Point spill responses
A technical report submitted to the Coastal Response Research Center by
Dr. Seth Tuler, Rebecca Kay, and Dr. Thomas P. Seager
-and-
17 April, 2006
NOAA Grant Number: NA04NOS4190063. Project Number: 05-983.
Social and Environmental Research Institute, Inc.278 Main Street, Room 404, Greenfield, MA 01301413.773.9955 www.seri-us.org
Dr. Igor Linkov58 Charles St., Cambridge MA 02141
Objectives and performance metrics inoil spill response: The Bouchard-120
and Chalk Point spill responses
S.P. TULER, R. KAY, T.P. SEAGERSocial and Environmental Research Institute278 Main Street, Room 404Greenfield, MA [email protected]@[email protected]
I. LINKOVCambridge Environmental Inc, Cambridge, 58 Charles StreetCambridge, MA [email protected]
1
OverviewOil spill response is defined as encompassing all activities involved in containing and cleaningup the oil in ways that achieve the following over-arching goals:
1. Maintaining safety of human life;2. Stabilizing the situation to preclude it from worsening, and3. Minimizing adverse environmental and socioeconomic impacts by coordinating all
containment and removal activities to carry out a timely, effective response.
Because so many different and interested parties are affected, there is the potential forconflicting ideas about how a response should be organized and implemented to achieve thesegoals. Moreover, one goal may create tension with another. There is room for differences ofopinion about what the over-arching goals mean in specific contexts and what objectives areassociated with each goal. We can expect then that assessments of interested and affectedparties about an oil spill response effort may differ because they have different objectives orbecause they evaluate them differently. In other words, they may select and apply differentperformance metrics.
This report addresses the question of what objectives and performance metrics are used by keystakeholders to assess two recent oil spill responses:
1. The Bouchard-120 spill response that began on April 27th, 2003 as the tugboat EveningTide ran aground and released No. 6 home heating fuel oil just at the entrance toBuzzards Bay, Massachusetts.
2. The Chalk Point spill response that began on April 7, 2000 when an intrastate pipelinethat transports oil to the Potomac Electric Power Company’s (PEPCO) Chalk Pointelectrical generating facility in Prince George’s County released No. 2 and No. 6 homeheating fuel oil into Swanson Creek and subsequently into the Patuxent River.
We conducted interviews with participants in each of the spill responses and investigatedpublished reports to gather information about:• roles and experiences of key responders and other interested and affected parties,• their concerns about spill impacts and response,• their views about and objectives during the response, and• their views about response performance metrics.
Those interviewed represented a diverse group involved in the spill response, including federal,state, and local government officials, NGO representatives, and local community members. TheFederal on-scene coordinator and State on-scene coordinators from each response wereincluded in the interview samples.
We used a grounded theory approach to inductively identify people’s objectives during the spillresponse. The objectives expressed during the interviews were grouped into categories. Wethen extracted all performance metrics expressed by the interviewees that were related to eachof the objectives. These performance metrics were then coded as:
• resource-based metrics: measures of the resources applied operationally to perform theactivities,
• process-based metrics: measures of the quality of activities performed to achieve thedesired outcomes, and
• endpoint-based metrics: measures of state variables relating to a desired outcome orobjective.
I
n addition, we compiled all interviewee comments regarding:• characteristics of “good” performance metrics• challenges to the use of performance metrics to plan and assess spill response
We did not evaluate any of the suggested metrics either according to the interviewees owncriteria for “good” metrics or appropriate uses or those that were identified in our literaturereview report.
We conclude the report with a discussion and comparison of findings from each of the casestudies and a discussion of how the interviewees define good metrics, how they understandtheir use and relevance, and challenges they identified to their use. The interviews provideuseful insights into the ways that different people involved with a specific spill response thinkabout characteristics of “good” metrics. An understanding of how participants involved in spillresponse think about “good” characteristics is critical for developing a process for collaborativelyselecting metrics prior to a spill event. Stakeholders must develop clear criteria on which toassess alternative metrics; their understandings can be informed by the literature onperformance metrics (which we review in an earlier report) and by an understanding of their ownviews. In addition, a number of challenges to using performance metrics for assessing oil spillresponse can be gleaned by the findings from this case study. Such challenges are importantto document in order to develop strategies for guiding performance metric selection in such away that the challenges are met. Pitfalls in performance metric selection and definition can beavoided by a careful understanding of these challenges; this is one of the features that ourproposed process for collaboratively selecting metrics prior to a spill event must overcome.
1.0 IntroductionOil spill response is defined as encompassing all activities involved in containing and cleaningup the oil in ways that achieve the following over-arching goals1:
1. Maintaining safety of human life;2. Stabilizing the situation to preclude it from worsening, and3. Minimizing adverse environmental and socioeconomic impacts by coordinating all
containment and removal activities to carry out a timely, effective response.
To achieve these goals, oil spill response efforts must include a variety of participants. Theycan include: federal, state, and local officials (i.e., Coast Guard, EPA, DNR, DOE, fire chiefs,harbormasters), the responsible party and its contractors, non-local clean-up crews hired byprivate contractors, environmental and community advocates at the national, state, and locallevels, and community residents who have an important stake in the response, (i.e., business-and home-owners, and beach associations).
Because so many different and interested parties are affected, there is the potential forconflicting ideas about how a response should be organized and implemented to achieve thesegoals. Moreover, one goal may create tension with another. There is room for differences ofopinion about what the over-arching goals mean in specific contexts and what objectives areassociated with each goal. For example, response strategies are based on the characteristics ofthe oil released, the location and timing of the release, and the kinds of resources in harms way.Response strategies are also dependent on the priorities placed on protecting specificresources in the context of a particular spill. Decisions made about priorities are to a very largeextent a political or social decision based on competing values. As one federal official noted,termination endpoints depend “on whose ox is being gored” (BI08). For instance, will shellfishbeds and/or frequented tourist beaches have more stringent endpoints than other less valuedresources or beaches? Will limited response resources be devoted to saving a much usedrecreational area or a sensitive marsh habitat even if the endpoints are unattainable or the lawof diminishing returns applies? As another federal official said, “There’s some value judgmentthere that somebody has to make somewhere along the line. And usually it has to do with theimpacts to people or acute mortality to birds, something that people might get upset about…wemade that decision …on a plover habitat…there was a point in time where we said, look, weknow that there’s still oil there. But, the clean-up itself is doing more damage than the oil isgoing to do” (BI09).
We can expect then that assessments of interested and affected parties about an oil spillresponse effort may differ because they have different objectives or because they evaluatethem differently. In other words, they may select and apply different performance metrics.Performance metrics are important because they can drive the response strategies and actions.They are also used to justify actions and obtain support after a response, from the public,government officials, and the responsible party (who must pay for the response and restoration).
This report addresses the question of what objectives and performance metrics are used by keystakeholders to assess two recent oil spill responses:
1. The Bouchard-120 spill response that began on April 27th, 2003 as the tugboat EveningTide ran its tanker aground and released No. 6 home heating fuel just at the entrance toBuzzards Bay, Massachusetts.
1. http://response.restoration.noaa.gov/book_shelf/910_response.pdf, pg. 9. See also
www.uscg.mil/d1/staff/m/rrt/
2. The Chalk Point spill response that began on April 7, 2000 when an intrastate pipelinethat transports oil from the Potomac Electric Power Company’s (PEPCO) Chalk Pointelectrical generating facility to residents in Prince George’s County, released No. 2 andNo. 6 home heating fuel oil into Swanson Creek and subsequently the Patuxent River.
Section 2 provides brief overviews of the Bouchard-120 and Chalk Point spills and responses.Section 3 includes a discussion of our research methods, including a description of our interviewsamples in each case. Through one-on-one interviews, we investigated the objectives of thesestakeholders and how they thought the response effort was or should be assessed with respectto these objectives. We did not evaluate the spill response, but, rather, identified how peopleinvolved would evaluate the response effort. They identified a variety of performance metrics aswell as their strengths and weaknesses as measures of performance. Finally, we learned whatthese stakeholders believe are appropriate uses of metrics in addition to characteristics of goodmetrics. Section 4 presents the objectives and performance metrics that emerge from theBouchard-120 case study, while Section 5 does the same for the Chalk Point case.
In Section 6 we compare the findings from both cases and discuss important themes thatemerge from the analysis, including challenges to using performance metrics and theinterviewees views about good performance metrics. The interviews provide useful insights intothe ways that different people involved with a specific spill response think about characteristicsof “good” metrics. An understanding of how participants involved in spill response think about“good” characteristics is critical for developing a process for collaboratively selecting metricsprior to a spill event. Stakeholders must develop clear criteria on which to assess alternativemetrics; their understandings can be informed by the literature on performance metrics (whichwe review in an earlier report) and by an understanding of their own views. In addition, anumber of challenges to using performance metrics for assessing oil spill response can begleaned by the findings from this case study. Such challenges are important to document inorder to develop strategies for guiding performance metric selection in such a way that thechallenges are met. Pitfalls in performance metric selection and definition can be avoided by acareful understanding of these challenges; this is one of the features that our proposed processfor collaboratively selecting metrics prior to a spill event must overcome. The report concludeswith a brief conclusion and discussion of how this research fits into the overall project plan.
2.0 Overviews of two oil spill responsesIn this section we provide brief overviews of the two cases that inform this research, theBouchard-120 and Chalk Point spills and responses. We begin by briefly describing the spillevent itself and then describe both the response effort and the effects of the spill.
2.1 The Bouchard-120 spill and responseOn Sunday afternoon, April 27th, 2003, the tugboat Evening Tide was towing the unmanned fuelbarge, Bouchard-120 (property of the Bouchard Transportation Company Inc.) from Philadelphiato the Mirant Power Generating facility along the Cape Cod Canal in Sandwich, MA. Theaccident occurred in Buzzards Bay (Figure 1). It is believed that the tanker, transporting No.6fuel oil, ran aground just at the entrance to the Bay trailing an oil slick, initially thought to be justsheen, in its wake. The vessel continued on course for ten miles before stopping, neverrealizing its cargo had been compromised (Figure 2).2
The initial call reporting the incident to the US Coast Guard (CG) was at approximately 5:30pmSunday evening. However, it was not until the early hours of Monday morning when officialsrealized the extent of damage to the ruptured cargo tank, a 12ft by 2ft gouge in the bottom ofthe hull estimating a leak between 21,000 and 98,000 gallons3 of oil from the vessels cargohold.
The ResponseWithin two hours of the reported accident, the CG arrived on site under the guidance of CaptainMary Landry, the Federal-On-Scene Coordinator4 (FOSC). In a matter of just a few hours more,various other state agencies such as MassDEP, the State-On-Scene Coordinator (SOSC), andfederal agencies arrived, as did the Bouchard Transportation Company and their contractedclean-up companies5. The SOSC was responsible for notifying the local first responders bytelephone about the spill. Other key official players involved in the response and clean-up werevarious local, first-response officials, and non-profit local advocacy groups, such as the Coalitionfor Buzzards Bay and the Buzzards Bay National Estuary Program.
2 http://www.buzzardsbay.org/ impacted areas.htm3 The initial reported amount of actual oil spilled was highly controversial. Independent Maritime Consulting Ltd., aprivate firm hired by the Bouchard Transportation Co., estimated a range of 21,000–55,000 gallons of oil leaked(http://www.buzzardsbay.org /volumespilled/volume-spilled-report.pdf). Other organizations, primarily the non-profit Buzzards Bay National Estuary Program, contested that the estimate was closer to 98,000 gallons, a quantitythe CG concluded as the best estimate (http://www.buzzardsbay.org/oilvolume.htm; http://www.buzzardsbay.org/volumespilled/bbpoilvolumerebutall. pdf).4 Unified Command (UC) is the federal coordinating body for organizing the containment, removal, and disposalefforts of oil during a spill. It uses a management system called the Incident Command System (ICS) whichincorporates federal, state, and local authorities in making decisions and carrying out actions related to the incidentplan. In the event of a coastal spill, the USCG is the FOSC responsible for organizing response actions. Infreshwater systems the duty of FOSC is assigned to the EPA (Costa, 2003). In addition, The NOAA ScientificSupport Coordinator (SSC), in accordance with the National Contingency Plan, provide the FOSC scientific advicewith regard to the best course of action during a spill response. The SSC performs a variety of tasks, includingworking to achieve consensus from the Federal Natural Resource Trustee Agencies, providing spill trajectoryanalysis data, and sharing information on the resources at risk, weather information, tidal and current information,etc. The SSC is the lead point of contact for the Scientific Support Team from NOAA’s Hazardous MaterialResponse and Assessment Division to the spill’s Unified Command staff (http://www.uscg.mil/d14 /units/msohono/ics/fog/ fog4520.htm).5 The Bouchard’s hired private contractors included Gallagher Marine, Clean Harbors, Entrix Environmental, Tri-State Bird, and Frank Corp. GEOInsight has been acting as the Licensed Site Professional overseeing the finalphases of clean-up.
The overall response consisted of five phases characterized in Table 1. Overhead flights wereinitiated to assess the situation followed by mobilization of response materials and people thatinitially focused on containing the oil around the barge with boom equipment. Skimmers andpompoms were additional equipment called in to recover the spilled oil. A temporary bird triagecenter was built on the grounds of New Bedford’s wastewater treatment facility, as specified perthe Area Contingency Plan (ACP). The triage effort was coordinated by Tri-State Bird Rescueand Research where they successfully mobilized quickly and coped with high volumes of birds,particularly loons. In subsequent days, strong winds and rough seas coupled with the extent ofmovement of the oil prevented the slick from being fully contained contributing to the landing ofoil dispersed to all areas of the Bay (http://www.buzzardsbay.org/impactedareas.htm). In somecases oil was “blown back” onto shores that had at first avoided oiling or had been recentlycleaned.
Table 1. Response phases, goals, tasks, and schedule
Phase Role Tasks Performed Duration/CompletionDate
Phase I Emergency/InitialResponse
Stabilization of vessel Stoppage of oil release Mobilization of response equipment Protection of environmentally sensitive
areas Initial cleanup of heavily oiled areas
Completed September2003
Phase II Ongoing Treatmentand Removal of Oil
Different clean-up methods applieddepending on shoreline types
Appropriate methods selected byreviewing level of oiling, environmentalsensitivity, and other factors
None specified
Phase III Monitoring andMaintenance
Confirmation that initial clean-upreached initial cleanup level
None specified
Phase IV Official Inspection Inspection of impacted, cleaned areas No area will pass inspection in this
phase unless decision of “sufficientlyclean” is unanimous by all authorities
Lasts several years
Phase V Long-term Cleanup Detailed and comprehensiveexamination and assessment of anyresidual impact
Questioning if the spill has damagedany natural resources
The Incident Command (IC) post location was initially located at the CG station on theMassachusetts Military Reservation; this location was chosen because it was initially thoughtthat the most oiling would impact the Cape Cod shoreline and the military base couldaccommodate all of the equipment necessary for an operational command center (i.e., phonelines, computers). However, local responders soon began to complain about this locationbecause of its distance from the most heavily impacted communities and difficulties with accessresulting from the need for security clearance for non-military personnel. The command centerwas relocated to the Library at the Massachusetts Maritime Academy. However, as a result ofthe distance between the actual spill site to IC headquarters, its location proved to be
problematic; it was changed yet again a month later, June 18th 2003, to its final position inMattapoisett at the Town Fire Station on Route 6.
Additional problems arose as well during the initial response. For instance, staging areas withstored response equipment had been identified in the ACP. However, pre-approval bylandowners was overlooked so certain gates accessing these areas were closed. Valuable timethat could have been devoted to response was lost because responders were trying to locatekeys for entrance into these areas. Communication within the first few days was also a problem.There was an overall lack in numbers of cell phones as well as radio frequencies available toresponders to facilitate coordination between and amongst them.
Summary briefings by various groups attending the IC meetings were held several times dailyinitially. This was diminished to twice daily once the situation was better under control. Inaddition, public meetings were held over the course of the response and clean-up in order tokeep the public abreast of UC’s activities. To date, the last public meeting was held at theWhaling Museum in New Bedford, MA in May, 2005, two years after the incident. The BuzzardsBay National Estuary Program maintained a website with information about the spill, includingsummary minutes of the command center briefings; this was one of the main avenues of publicinformation during the initial response effort and it was highly commended by federal officials.
After the initial efforts to contain the oil, the response turned to the protection of key areas andspecies that were identified as sensitive. Ram Island (Mattapoisett, MA) and Bird Island (Marion,MA) were two such locales that the ACP identified as high priority to protect because of nestingendangered roseate tern habitats located there (Figure 3). In the Ram Island case, boom wasdeployed in order to prevent oiling of the Island. However, no persons remained on the scene inorder to monitor the effectiveness of the boom. The booming effort failed and the Islandsustained a substantial amount of oiling before the failure was realized. Great efforts were thenmade to prevent the birds from inhabiting and nesting on the island by using an innovativetechnique of hazing while clean-up crews worked to remove the oil from the rocks via manualand power-washing. In the end, only one roseate tern mortality was observed from theimmediate effects of the spill.
Other fragile areas, like Allen’s Pond in Dartmouth, MA were bulldozed and sandbagged inorder to close off certain channels so that particular inlets were protected from being impacted.Response, and ensuing clean-up, within the Buzzards Bay community continued focusingprimarily in the more heavily impacted towns listed here: New Bedford, Mattapoisett, Dartmouth,Fairhaven, and Marion (see Figure 1 above).
The EffectsThe conspicuous nature of No.6 oil6 coupled with the complex craggy Buzzards Bay coastlinemade clean-up a challenge. Therefore, on June 27th, 2003, a clean-up plan, the ImmediateResponse Action Completion (IRAC), was prepared in order to determine which shorelinesneeded imminent clean-up. Table 2 is a classification of priorities.
1A Heavily Utilized, Public Recreational Sand Beaches1B Less utilized, Semi-Public and Private Sand Beaches1C 1C. Mixed Sand and Gravel, Gravel (pebble to boulder) and Rip Rap Groins (jetties)1D Rip Rap Seawalls, Bulkheads, Piers, Docks and Pilings1E Rocky Shorelines (bedrock outcrops)1F Salt Marshes2 Roseate Tern Habitat3 Piping Plover Habitat(http://www.buzzardsbay.org/oilpics/cleanupplan6-27-03.pdf)
There were a variety of impacts, socio-economic and ecological, from the Bouchard-120 spill(Table 3).
Table 3. Ecological and socio-economic impacts from the Bouchard-120 spill
Species Impacted Socio-economicLoons Beach closuresRuddy Ducks Summer cottage rentalsRoseate Terns Tourist RevenuesPiping Plovers Lost fishing daysClams & Quahogs Shellfish bed closuresLobstersTiger Beetles
2.2 The Chalk Point spill and responseOn April 7, 2000 an intrastate pipeline that transports oil from the ST Services storage terminalin Aquasco, MD (St. Mary’s county) to the Potomac Electric Power Company’s (Pepco) ChalkPoint electrical generating facility (Prince George’s County) released upwards of 100,000gallons of No. 2 and No. 6 home heating fuel oil into Swanson Creek and subsequently thePatuxent River. The spill has been recorded as the worst environmental disaster in Maryland’shistory affecting wildlife, natural resources, and property owners.
According to the National Transportation Safety Board’s (NTSB) 2002 immediate press release,the cause of the spill was due to a fracture in a wrinkle in the pipeline that was not discovereddue to an inaccurate reading of an in-line inspection device. The Board discovered in their
6 While No. 6 fuel oil is not as acutely toxic to organisms in the water column like No. 2 oil, it is very challenging toclean-up due to its thick and viscous nature and its tendency to float. It is potentially harmful to wildlife, primarilybirds, because of their propensity to dwell on the water’s surface and it is associated with adverse effects due tolong-term sedimentation contamination (http://www.buzzardsbay.org/number6oil.htm).
investigation that the leak occurred in the morning of the 7th, yet was not discovered oraddressed until the late afternoon. In addition to the delay in detection, Pepco reported to theNational Response Center (NRC) that 2000 gallons had leaked from the pipeline. However,once EPA officials responded that evening, it was realized that in actuality the figure was closerto 125,000 gallons. Some emergency responders believed that Pepco’s imprecise reporting ofthe spill’s magnitude could have potentially impeded the response efforts with adequatepersonnel and equipment and the evaluation of the spill’s threat to the area.
The ResponseWhen the spill was reported to the federal authorities there was some initial confusion as towhose authority it fell under, the EPA’s or the Coast Guard’s. Once it was determined that it wasin fact in EPA jurisdiction, Colby Stanton was identified as the FOSC; likewise, the SOSC wasAlan Williams from the Maryland Department of the Environment (MDE). Other key officialplayers involved in the response and clean-up were Maryland Department of Natural Resources(DNR), US Fish & Wildlife Service, Tri-State Bird Rescue and Research, and various clean-upcontractors hired by Pepco. In total, there were 12 local, state, and federal agencies respondingon the scene.
Calvert, Charles, Prince George and St. Mary were the counties most affected by the spill.Recovery methods were implemented in the Swanson Creek area via deployment ofcontainment booms in order to prevent the spread of the slick. Unfortunately, like the Bouchardincident, weather conditions worsened resulting in strong winds, choppy tidal conditions, andexcessive rain, all negatively impacting containment efforts. Such circumstances caused the oilto breach and crest over the booms, spreading into the Patuxent River and its tributaries. Themajority of the oil was spilled in Swanson creek, but the adverse inclement weather spread theoil 17 linear miles downstream from point of entry and oiled 40 miles of shoreline. Remediationefforts continued heavily for weeks after the spill and in June, fish and shellfish were declaredsafe for consumption.
The EffectsThere were a variety of impacts, socio-economic and ecological, from the Chalk Point spill.Table 4 lists some of the more predominant ones.
Table 4. Ecological and socio-economic impacts from the Chalk Point spill
Two years after the spill at the former Pepco Chalk Point plant, NOAA, the U.S. Fish andWildlife Service, the Maryland Department of Natural Resources and the Maryland Departmentof the Environment unveiled a $2.7 million plan to restore the damaged natural habitat. Thegovernment plans to hold Pepco accountable and responsible for the restoration of the area to"pre-spill conditions.” In addition, they are required to compensate local residents for their lossof recreational opportunities. The plan calls for projects such as creating oyster reefs in thePatuxent River, beach habitat for diamondback terrapins, and restore nesting habitat hundredsof miles west in the Midwest for the migratory ruddy ducks who were caught in SouthernMaryland during the spill (McCaffrey, 2002). More than 100 fishermen filed claims, as well asthe owners of about 400 properties. (McCaffrey, 2002)
3.0 Research methodsThe purpose of the case study is to better understand how people assess or judge performanceof spill response efforts in particular cases. Understanding how people assess spill responseefforts requires us to also understand their goals and objectives in oil spill response. It turns outthat it is also important to understand the ways in which response actions are related to damageassessment and restoration related activities.
To explore these issues in the context of the Bouchard-120 and Chalk Point spills andresponses we conducted a series of interviews and investigated published reports to gatherinformation about:
• roles and experiences of key responders and other interested and affected parties,• their concerns about spill impacts and response,• their views about the response, and• their views about response performance metrics.
The interviews were designed to be semi-structured and open-ended. The interview guide isprovided in Appendix A. The interviews were not designed to systematically elicit informationabout relative priorities among objectives or performance metrics. Instead, at this time, we weremore interested in learning about the range of objectives and metrics among key participants.Moreover, we gathered information about the types of metrics that interviewees believed wereused or should be used, as well as those that were not or should not have been used to assessthe response effort. Our focus in these case studies was also on gathering information frompeople who were actively engaged in each of the spill responses.7 Both of thesemethodological choices affect the data we gathered and our interpretation of them. Again, ourinterest was in the range of metrics that could be applied to assess the response effort – thisinformation is what we need to perform subsequent project tasks.
We conducted 12 interviews for the Bouchard-120 case study. In one instance two individualswere interviewed together. All interviews were conducted by either Seth Tuler or Rebecca Kayand lasted between 1 to 1.5 hours. Those interviewed represented a diverse group involved inthe spill response, including federal, state, and local government officials, NGO representatives,and local community members. The Federal on-scene coordinator and State on-scenecoordinators were included in the interview sample. Specific organizational affiliations included:
• US Coast Guard• NOAA• FWS• Buzzards Bay National Estuary Program• MA DEP• Town fire departments• Town Harbor Masters• Town Environmental Affairs Coordinator• Contractor for the responsible party
We conducted 18 interviews for the Chalk Point case study. In one instance two individualswere interviewed together. All interviews were conducted by Rebecca Kay and lasted between 1to 1.5 hours. All except one were recorded digitally. Those interviewed represented a diversegroup involved in the spill response, including federal, state, and local government officials,
7 We will have a chance to gather similar information from local residents in areas impacted by the Bouchard-120spill as part of Rebecca Kay’s masters thesis research that is being conducted this spring and summer (using focusgroups and interviews).
NGO representatives, and local community members. The Federal on-scene coordinator andState on-scene coordinator were included in the interview sample. Specific organizationalaffiliations included:
• EPA• MDE• MD DNR• DOT• Homeowner• MD Board of Public Works administrator• MD Department of Housing and Community Development• FWS• NOAA
Our approach to identifying the interviewees was a snowball sampling technique. In snowballsample participants are not selected to be representative of a random selection of a populationbut instead are meant to represent cogent perspectives. We selected people that representdifferent points of view, have well-formed opinions based on their experiences with the spill, andwere willing to be interviewed. We identified these people through our review of documentsabout the spill (i.e., official reports, newspaper articles) and suggestions by others. During eachinterview (or preliminary conversation with a potential interviewee) we also asked: “who elseshould we be talking with?” In this way, our interview sample was built as we went – this is the“snowball” approach.
In both cases we were unsuccessful in contacting some individuals we wanted to interview orwe contacted people who declined to be interviewed. For example, in the Bouchard-120 casesome local environmental advocates declined to be interviewed because they were involvedwith litigation against the responsible party. Representatives of the responsible party and itscontractors similarly declined. In the Chalk Point case we were unable to make contact withsome Coast Guard staff. Finally, we made limited attempts to contact and interview localresidents from communities affected by the spills. Our intent in these case studies was togather views and information from people who were actively engaged in spill response. Whilelocal residents may have experienced harm, been worried about potential harm, or participatedin public meetings, they did not actively participate in spill response actions.8
A limitation of a snowball sampling approach is that it does not provide information about thefrequencies of viewpoints or responses. If done well, it elicits information about the range ofviewpoints. As it relates to this project, that would be about the goals and concerns related tospill response, criteria for judging the quality of the spill response (i.e., performance metrics),and observations about the use of metrics.
A summary of each interview was written. Then, each digitally recorded interview was listenedto (sometimes multiple times) where we then coded the interviewee’s responses, as describedbelow. We did not prepare verbatim transcripts of each interview in its entirety, but some limitedtranscription was made of key parts of selected interviews.
8 Rebecca Kay’s masters thesis research is focusing on the opinions of local residents about spill response andcommunication efforts. She will be conducting interviews and/or focus groups with residents of communitiesimpacted by the Bouchard-120 spill during the spring and summer. Her thesis is due to be completed by the end ofsummer 2006.
With many hours of recorded interviews, it was necessary to identify the information relevant toour guiding questions and interests. Towards this end, we used a grounded theory approach(Glaser and Strauss 1967). Grounded theory is a qualitative research tool. Moore (1996) andTuler and Webler (1998) used this approach in their study of participants' definitions ofsuccessful outcomes in participatory decision-making. In grounded theory, important conceptsemerge inductively during the data analysis rather than in advance of the investigation. Dataare categorized with respect to relevant similar characteristics in a process called “coding.” Atfirst, a relatively large number of categories are developed. Then, through iteration thesecategories are grouped into more abstract categories of conceptual relevance to the analysis;data and categories are grouped according to their relationships with each other. For example,all statements related to "reducing bird injuries" or "protecting nesting habitat" can be groupedinto a category named "response should protect bird populations." The primary reason forchoosing this methodology is that it seeks to minimize researcher influence and instead letsstakeholder concerns and possible performance metrics emerge directly out of the datathemselves.
Approaching data analysis in this way requires the researcher to collect, code, and categorizedata while always reassessing and remaining sensitive to the changing dynamics of thefindings. The process of inductively generating principles in the course of the fieldworkdiminishes the influences of pre-conceptions and assumptions which can direct the resultsalong lines predetermined. At the same time, data collection and analysis in a grounded theoryapproach requires a great deal of judgment. We remain constantly vigilant of the danger of aself-fulfilling prophecy by reminding each other to step back and re-examine our analysis for anysigns of bias. Throughout the project we adopted an attitude of friendly, critical re-examinationfor the purpose of maximizing our theoretical sensitivity, a term that refers to the ability ofresearchers to complete these activities in a manner that remains true to the meanings thatemerge from the data (Glaser 1992).
Whenever there is more than one individual involved in coding, inter-coder reliability canintroduce errors. We have had to ensure that there is a reasonably high likelihood that thesame text would be coded in the same way by Tuler and Kay (the two primarily responsible foranalyzing the interviews). We took several standard measures to ensure inter-coder reliability.First, we coded one interview multiple times, each time comparing and discussing ourinterpretation of the interview. This allowed us to refine our coding sheet and develop a clearerunderstanding of how to interpret the interviews. Second, we each coded another interviewtogether, discussing our coding as we went along. Except for these two, each interview is beingcoded only by one person. We continued to meet in order to discuss our coding process. Bydiscussing differences, we were able to attain better clarity and agreement. We also hadfrequent conversations about the codes and principles for assigning codes. Finally, we re-codedinterviews when subsequent discussions revealed misunderstandings about how the two of uswere interpreting them. We have found these procedures to be effective in past researchprojects.
After coding the interviews we grouped the identified spill response objectives into moreabstract categories; this is referred to as axial coding in a grounded theory framework. In thisway common themes among the coded objectives are identified. We then extracted allperformance metrics expressed by the interviewees that were related to each of the objectives.These performance metrics were then coded into the following categories, as described in theliterature review report:
• Resource-based metrics: measures of the resources applied operationally to performthe activities,
• Process-based metrics: measures of the quality of activities performed to achieve thedesired outcomes, and
• Endpoint-based metrics: measures of the state variable relating to a desired outcome orobjective.
In addition, we compiled all interviewee comments regarding:• appropriate uses of performance metrics for assessing oil spill response efforts and• characteristics of “good” performance metrics.
We did not assess any of the suggested metrics either according to the interviewees owncriteria for “good” metrics or appropriate uses or those that were identified in our literaturereview report.
4.0 Objectives and performance metrics in the Bouchard-120 responseIn this section we describe our main findings related to the Bouchard-120 case study.First, through review of written documents and our coding of interviews, we identified from theinterviewee’s perspective, the important objectives of this spill’s response. We grouped theseobjectives into several categories.
Then, we identified the metrics that individuals used to assess the quality of the spill responseby asking, “Was it a successful response or not? Was it a ‘good’ response or not? How is thatjudgment made?” Metrics, by their very definition, are used to assess progress toward specificgoals or objectives. We provided, in tables, the metrics identified by our interviewees associatedwith each objective. In some cases a metric was associated with multiple objectives.
Finally, we discussed what people told us about the quality of proposed metrics. In ourinterviews we asked that people explain to us what metrics were used (or ones they have seenused), ought to have been used and which ones they prefer. This provided them an opportunityto reflect on the quality of particular metrics and the difficulties of applying them in any sort ofstrategy development, decision-making, or assessment of the response. These challenges andlimitations are discussed as well. We do not indicate for all metrics whether or not intervieweescharacterized them as “good” or “bad” – this is being left to future project tasks, although we doprovide some examples of what they said.
We found that many different objectives for the response to the oil spill were important to ourinterviewees. We grouped them into the following broad categories:
• Protect worker and public health and safety• Protect environment/ecological systems• Establish a coordinated and effective response framework• Implement an effective and timely response• Mitigate economic impacts• Meet legal and regulatory requirements• Mitigate social nuisance impacts• Address needs and concerns of the affected public/communities• Gain public support for the response
Each of these is discussed in the following sections.
Protect worker and public health and safetyThe protection of worker and public health and safety is the first goal of the incident commandsystem.9 The priority of health and safety is reflected at all levels of government involved in theresponse.
While protection of health and safety is a priority, there are difficulties in assessing theresponse’s effect on health and safety, particularly from exposure to the oil and its byproducts.For example, one federal official noted the difficulty associated with testing for potential healthrisks:
• Once you close [a fishery] for oil it becomes a real big problem because there aren’t awhole lot of standards for oil. The reason there are not a whole lot of standards for oil isthat the PAH level in a shellfish caught here [Boston]… will be real, real different from
what’s caught off the pristine waters off of Buzzards Bay; But they don’t want to closethese guys [Boston] down and they don’t want to give these guys [Buzzards Bay]permission to pollute more. So you can’t have this state standard so the backgroundsare real different; it’s kind of a catch 22. Once you close it for oil you have to re-open itunder some premise which is really hard to do and it has to be defensible. You also getthis whole new group of people [state health workers] coming in who typically don’t dealwith oil…Instead of looking at it as oil, they start breaking it down into its constituents…[and they require that the different components must reach the standards for thatcompound]… (BI08).
Interviewees identified very few performance metrics that could be used to assess the degree towhich health and safety were achieved, as shown in Table 5, in spite of its importance as agoal. This may be a reflection that counting injuries and fatalities is relatively easy and is themost direct performance metric.
Table 5: Performance metrics related to protecting health and safety identified ininterviews
Response performance metricsEnd-Point • Number people killed or injured
• Number of mishaps during hours worked
Process •
Resource • Number of IRAC team members OSHA/HAZMAT trained
Protect environment/ecological systemsMany interviewees spoke of the importance of protecting ecosystems. In some cases theirfocus was on habitats while others expressed concern about specific species (at bothpopulation and individual scales). Within this broad category individuals identified severalspecific objectives.
First, individuals wanted to prevent and mitigate impacts to critical habitats or areas. Forexample:
• It was Buzzards Bay and we had identified this as a really sensitive area…this was oneof those areas that was designated as pretty important to get on response strategiesearly. (BI09).
Second, many commented on protecting endangered and threatened species (not habitats),particularly bird species:
• From a resource impact perspective, there were two key endangered species; one theroseate tern and the other the piping plover, which is actually a threatened species andit’s a big deal around here…and the tiger beetle, an endangered beetle (BI08)
• We had tern islands, plover beaches and salt marshes…they are endangered specieshabitats. (BI09)
In some cases their concerns were expressed in terms of specific species (e.g., loons, RoseateTerns) or individual birds and the objective being to prevent or mitigate impacts to them. Insome cases, their concern was expressed at the population level:
• generally speaking, we consider the endangered [Roseate] adults more important toprotect than one years worth of reproduction because the adults can come back next
year and reproduce…your chances are better of getting productivity if you are protectingthe adults (BI09).
Third, others identified the importance of protecting a variety of species, not all of which werelisted as endangered or threatened, including invertebrates in inter-tidal zones and shellfish.
• Barney’s Joy has marsh grass, it has a lot of shellfish resources, it’s a very undeveloped,very pristine, it’s a sandy recreational area…it’s a sort of cross-section of resourcevalues that are present throughout Buzzards Bay (BI07).
Fourth, some individuals expressed the objective of protecting shorelines.• If the oil had come up on the shoreline there, they (the terns) would have gotten
whacked (BI09).
Fifth, avoiding worst-case scenarios was critical to some individuals we interviewed, assuggested by the following quote:
• “Worst hit” as a combination of volume and sensitivity of the resource that was hit, forexample Barney’s Joy (BI07).
Finally, a number of individuals identified the mitigation of ecological impacts from clean-upactivities as an important objective of spill response activities. Clean-up activities can lead toerosion, for example, and:
• Make sure that during clean-up they (the tiger beetles) didn’t get trampled…this wasmainly a response type of concern…clean-up people were able to clean-up the island toa sufficient degree that they were able to stop hazing and let the terns come down andfinish their nesting season (BI09)
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which protection of ecological systems was achieved, as shown in Table 6. Metrics based onmeasurement of outcomes, or end-points, were identified, as well as those that are related tothe resources applied to achieve those end-points and the activities (processes) performed toachieve them. The end-points measured are rarely directly about the health or quality of theecological system or its components. In some cases the metrics assess indirect or interimindicators of the state of an ecological system, habitat, or populations. For example:
• How many pounds of debris, oil contaminated debris, has been recovered and sent offfor disposal? (BI02)
• Number of miles of shoreline impacted depends on the degree that its impacted, thetype of shoreline, the environmental sensitivity of that shoreline, and the socio-use ofthat shoreline (BI10)
• How many birds were oiled, how many recovered, how many released, and how manyreleased and survived? (BI02)
• How many acres of shellfish areas were closed and then re-opened? (BI02)
Table 6: Performance metrics related to protection of environment/ecological systemsidentified in interviews
Response performance metricsEnd-Point • No re-oiling or residual oil causing chronic toxicity to something
• Number of estuaries “shut off” (i.e., areas prevented from oiling from via booming)• Number of inlets that were able to boom off• Number of miles of shoreline impacted• Amount of oil or globules on shore• Number of areas where prevented oil contamination• Oil contamination is on sandy beach (easier to clean-up than rocky inter-tidal zone)• Shellfish beds protected• Number of miles that have been cleaned to an acceptable level• How many pounds of oil contaminated debris has been recovered and sent off for
disposal?• How many gallons of oil have been recovered?• Number of birds oiled?• Number birds released and survived• Mortality/survival of wildlife (birds)• Number roseate terns lost• Number dead birds per unit search area• Number dead birds.• Percent birds rehabilitated• Number of birds rehabilitated• Have clean-up standards for shoreline contamination been meet
Process • Did getting required permits delay response action?• Booming deployed around critical habitat (Roseate tern nesting habitat, salt marshes)?• Attempt to direct oil to “sacrificial areas”?• Good choices made about where to deploy boom?• Oil being captured in open water before it hits the beach?• Proper placement of skimmers?• Resources placed in the proper locations• How fast [# of days/ # of hours] rehabilitation efforts brought in and set up operations• Is rehabilitation center handling all live birds coming in?• Speed at which able to handle birds coming in for rehabilitation?• Amount of area covered in search and recovery• Volunteers able to get plugged in well?• Time it takes to implement tasks, such as boom deployment?
Resource • Number sandbags deployed?• Number people on cleanup crews to deal with oiled beaches?• Amount of boom deployed• Number of floating resources to pick up oil in open water oil?• Number skimming units available and operable• Number of volunteers• Number of bodies to manage different aspects of response, including organizing
SCATs?• Is there a ‘bird searcher’ on each team?
Mitigate economic impactsUnderlying many of the concerns expressed in the interviews was the objective of minimizingeconomic impacts from the spill through the response. Federal officials talked of their fiduciaryresponsibility to minimize costs associated with the response and damage resulting from thespill:
• We’re serving the public as a civil servant so if they’re happy with the response, thenwe’re probably doing our job; As trustees, we’re serving the resources that arearound so if we’re mitigating damage then we’re doing a good job; as citizens, wehave a fiduciary responsibility not to spend money in an unnecessary way (BI08).
In addition, an objective was to minimize spill response costs. These include direct costs from,for example, costs of equipment and crews, and indirect costs, from, for example, recreationalbeaches damaged from clean-up activities (e.g., vehicles on beach).
Within this broad category individuals identified several other specific objectives.
First, mitigate economic impacts from lost recreation. Such losses can arise from fewer peoplevisiting contaminated beaches or because they are afraid that beaches are contaminated.Similarly, recreational fisheries may be affected both onshore (e.g., clams, striped bass) and off-shore.
Second, and related to the first, is the objective of mitigating economic impacts from losttourism. These effects include direct and indirect financial losses associated with fewervacation/summer rentals. Such effects can result from fewer and shorter visits from out-of-townvisitors and can extend to losses from visitors not eating in local restaurants, shopping at giftstores, etc.
Third, there was a concern with reducing the economic impacts to local commercial fishermen.Losses can arise from lost fishing days due to shellfish bed closures because of contaminationissues:
• If fishermen aren’t able to work … I think [that getting them back to work quickly is a]valid measure of success (BI02).
• The shellfishing issue was enormous because of the closure to beds…the fishermenwere troubled for a long time (BI03)
However, one person noted how such losses can also be associated with fears of contaminationaffecting market prices and consumption patterns:
• It’s almost always a taint issue, almost always a market issue and not a health issue [forshellfish] (BI08).
Fourth, mitigate economic impacts to towns from costs of clean-up. Towns incur costsassociated with local officials (e.g., harbor masters) working long, extra hours during aresponse. In addition, local equipment can be used. It was important to some interviewees thatlocal towns be compensated for extra work hours dedicated to spill response efforts:
• We kept track and were ultimately reimbursed, Bouchard paid for that (BI03).
Finally, there was also an interest to minimize the costs to the responsible party resulting fromthe response. For example, there is a “need to be fair to the responsible party. Don’t over-burden responsible party.”
Another person spoke of this being an objective that might conflict with other important goals.When gallons spilled is used to determine size of initial response, it can result in too slow of aresponse:
• Because they were unsure about how many gallons had been spilled, the spin-up gettingthe clean-up contractors there on-scene started out pretty slow. Again, the CG is prettyresponsive to the RP and they don’t want to mobilize an entire army if it turns out thatthe spill isn’t as big then you’ve spent all this money when you didn’t have to…[later] theCG admitted that, that the response spin-up time was actually slower than it should havebeen (BI09).
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which economic impacts were mitigated, as shown in Table 7. Metrics based onmeasurement of outcomes, or end-points, were identified. Interestingly, for this objective,measures that are often related to resources, such as costs of the response and laboratorywork, are here related to end-points.
Table 7: Performance Metrics Related to Mitigating Economic Impacts Identified inInterviews
Response performance metricsEnd-Point • Lost rental income?
• Lost tourism dollars?• Loss of summer rentals?• When beaches put back in use?• Recreational opportunities that were lost that are now back to what they were?• Acres of shellfish areas closed?• How many acres of closures reopened?• Duration of shell fish bed closures?• Acres of shellfish beds lost?• Costs of laboratory work?• Money spent on response?
Process • Kept track of all costs?• Towns reimbursed by RP?
Resource •
Mitigate social nuisance impactsA concern with aesthetic issues was identified by a few of the interviewees. They primarilyframed this concern about the way perceptions of success were mediated by people’sobservation of staining and experiences with unpleasant odors. For one person addressing suchsocial nuisance impacts was important because:
• That’s what connects people to the resource and imbues them with more of astewardship ethic… Aesthetic harm has a long-term impact and may reduce thestewardship ethic of people who use those environments for recreationalpurposes…and that aesthetic impact should not be discounted (BI07).
But as one federal official noted:• It is interesting to deal with the public; their perception of what’s bad versus what the
responders think is bad; ya know some folks want every speck of oil off their docks, theirboat…. (BI09).
A local official made a similar observation:• Success from the point of view of aesthetics, there are people who will see, and
probably will see for decades, the globs of oil stained on the rocks that could not becleaned off; they may say that it’s not successful because you can still see evidence ofthat (BI03).
Our data do not allow us to verify such assertions about “the public.”
Interviewees identified only a couple of end-point related performance metrics that could beused to assess the degree to which social nuisance impacts was achieved, as shown in Table8.
Table 8: Performance metrics related to mitigating social nuisance impacts identified ininterviews
Response performance metricsEnd-Point • Presence of stained rocks that lead people to say that it’s not successful
because you can still see evidence of that?• Complaints about stained rocks?• Oil on the beach?
ProcessResource
Establish a coordinated and effective response frameworkMaking the best possible decisions with limited information, uncertainties, and time pressureswas identified as an important objective by many interviewees. While they found it difficult toarticulate specific measurable performance metrics for them, it is nevertheless important as theyview good decision making to be closely associated with good outcomes. Within this broadcategory individuals identified several specific objectives.
First, it was important to gather accurate and relevant information for decision-making. Forexample, we were told that it is important to know the number of gallons and type of oil spilledas this dictates the amount of manpower and different types of response equipment needed fordifferent sorts of oil and their amounts spilled. Responders must determine what responsestrategies to choose. However, while others agreed that this is important information to have, itis not always helpful:
• In some instances it is [useful; it’s just that there was so much uncertainty as to howmuch was spilled… from an NRDA side, we’re not so much concerned with the gallonsspilled, but the impacted resources… sometimes the gallons can give you an idea as tothe range of impact; it’s not always the case. Mainly we tend to take our cues from theresponse in terms of the initial reaction from the NRD people (BI09).
In addition, we were told that other issues like weather conditions affect the decisions made:• Within a matter of a mile, we can have a wind shift takeover and not just a gentle
takeover… the weather bureau will say gentle NW and you’ll look out there and say,“there’s nothing gentle about this.” In regards to the weather bureau, local phenomenathey can not predict. There are local issues and environmental issues that makebooming a nice catch phrase, 25 mile/hr winds there’s not a boom in the world that’sgonna stop it (BI05).
Second, it was not only important to have accurate and relevant information, it was important tohave it quickly:
• It was not a matter of what information did I need that I didn’t have; it was probably morea matter of what information is available in a timely manner?...it was more a matter oftrying to get that information when you needed it (BI02).
Third, an aspect of “getting the right information” was the need to integrate localresponders/leaders into response planning because of their knowledge of local conditions,resources, etc.:
• Local municipalities really have a strong say and they want to be a part of the response,and they want information and they want to know what their role is…They did not feelthat they were getting information from either the state or federal government and they
felt that the information that they had that they were trying to get to us was not beingheard (BI02).
• There was a bit of a disconnect initially between the municipal response and theresponse being coordinated by the Command Center; it did not really adversely affectthings in a way it may have assisted because we were on scene, we had already hit thebeaches…by the time the response crews were really fully mobilized, they recognizedthat there were municipal people already onsite who kind of knew what was going on orwho knew where…we needed to do the response… Although we weren’t inside thatincident command system initially, they recognized that we were there and a resourcethat could be of assistance so we were integrated pretty quickly (BI03).
Fourth, it was also important that lessons from past spills and responses be integrated intodecisions and planning for this spill:
• We tried to learn from what we saw happen in RI… because [public] was not happy tobe left out of the process. They weren’t giving them anything to do so basically…theycriticized the response in the media. We paid attention to that and decided that we weregoing to try to avoid that. (BI02)
Fifth, strong and flexible leadership is viewed as important and some interviewees associatedgood decision making with strong and flexible leadership. Leadership can ensure that responseand damage assessment concerns are integrated, for example. Flexible leadership is importantto ensuring that good decisions are made and local resources used effectively when conditions,such as weather, change.
Difficulties of gathering accurate and relevant information were identified by some intervieweeswho point to problems of measurement of performance metrics. For example:
• If there’s one thing that would help me a lot is to…make sure that since both responseand NRD are getting there hands on the same types of data at least if we are collecting itin a way that’s going to be consistent and helpful to both sides I know that there issomeone responsible in incident command for collecting all the data…but what I am notsure happens is that someone looks at the forms and provides some feedback orcorrections to the SCAT teams themselves as encouragement to make sure all theblanks are filled out, etc...Spaces on the form are not always filled out and that doesn’thelp me (BI09).
Sixth, coordination among participating government agencies, contractors, etc. is important.Coordination affects the quality of plans and decisions (i.e., obtaining the right information in atimely manner) and how well plans are implemented and this reflects a concern withcoordination within the command center and between local responders and the commandcenter. For example:
• If there is an emergency and its not crime related, they (fire chiefs) are the public safetyofficers for the community. In Massachusetts if it has a direct public safety concern, theyare in charge over the state... Some of them were angry because they didn’t get enoughnotice ahead of time and were not brought into the process early on (BI08).
Seventh, as suggested earlier it is important to effectively involve local people. This includesfirst responders, such as harbor masters, fire chiefs, and environmental officers. However,interviewees also highlighted a concern with how volunteers could be included effectively:
• Another issue we have on the state level is volunteers. Volunteers… want to help…itstheir beach (BI02).
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which establishing a coordinated and effective response framework as shown in Table 9.
Table 9: Performance metrics related to establishing a coordinated and effectiveresponse framework identified in interviews
Response performance metricsEnd-Point • Modelers able to get 24 hours ahead of spill with accurate projections
• Are pre-identified areas potentially being affected?• Did we have information to keep Governor’s office and other state Senators and Reps
abreast of what was going on• Understanding of whether oil is still stored offshore re-contaminating cleaned up
beaches?• Is oil coming on shore several days later?• Accurate accounting of volume oil spilled?• Accurate accounting of amount of oil on the shore?• How many areas have been cleaned as of today?• Number of miles of shoreline impacted right now?• Bad feelings amongst locals responders toward the unified command staff?• Level of conflict about plans and activities?
Process • How quickly decisions made?• Chain of command clear?• Resources placed in the proper locations• Are decisions correct (in hindsight)?• Location of command center convenient and easily accessible?• Are crews visiting hard hit areas every day?• Shoreline assessment teams record information in a unified way 1) where is oil, 2) how
much, and 3) what does shoreline look like; want minimal variability• Organized, systematic way of recording where oil has come ashore and impacted is used• Systematic, ‘non-political’ approach used to deploy clean-up crews• Presence of watchdog to see what’s going on?• Is there conflict or chaos in command center?• Are players familiar with each other? and reliable communication technology available
(e.g., cell phones working)?• System established to track progress?• Experts consulted for input on response strategies?• Command center set-up and people mobilized?• Plans communicated day in advance?• Ability to effectively and timely monitor what happening in field?• Are there clear protocols?• Are there clear schedules?• Time it takes to implement tasks, such as boom deployment?• Is all pertinent data gathered and recorded?• Local responders notified quickly?• Conflicts and heated emotions among responders addressed?• All goals worked on together?• Attend to short, medium, and long term needs simultaneously?• Follow ‘best response’ protocol and integrated command system?• Revise objectives and activities based on monitoring effort?• Coordination of volunteers performed?• Volunteers integrated into clean-up appropriately?• Volunteers integrated into clean-up quickly?• Equipment and personnel demobilized when no longer needed?
Resource • Number of teams of trained observers walk coastline and make observations of extentand coverage area of oil
• Number crews trained• Type of oil• Phones available and working• Number of hours worked• Number supervisors out in field
Implement an effective and timely responseA critical objective the interviewees expressed was that response should be effective and timely.In other words, the response should achieve the established goals and do so quickly in order tominimize impacts from the spilled oil. While this set of objectives is very outcome oriented,metrics to assess them are also related to process and resources. For example, oneinterviewee’s assessment of the response was based on answers to her questions: Is theresponse being implemented effectively and in a timely way? Is the response quickly started?How quick? How well organized? (BI08)
Within this broad category individuals identified several specific objectives.
First, resources should be adequate and used effectively. They wanted to know if there was, forexample, a clear and efficient method of deploying people into the field:
• They were one of the best communities to work with because they weren’t breaking uppersonnel, and that is what happened in [other towns] –some people who worked in alltowns said it was a pleasure to work in [name of town] (BI05).
Similarly, it was important that:• there need to be enough people in the field (BI10);• the process be ramped-up to full level quickly (BI03);• an appropriate balance should be achieved in response between size and effort and
marginal gains/costs to RP (BI109);• resources for response should be matched to the kind of response needed (BI10).
Second, actions should be well-planned and implemented. This is related to good decisionmaking, but here the emphasis is on the effectiveness of plans and their implementation. Forexample, interviewees were concerned about:
• The crews did exactly what was said in the plan… and then thought, ok, we’re done withthis task, let’s go do something else. They should have maybe left a monitor there toactually see, that “ok this isn’t working. We better try to hurry up and do something else”(BI09).
• [The endpoints designed] were protective of fish and wildlife for this spill. I was satisfiedwith the cleanup endpoints…since we had actual definitions of what happened or hadnot happened on a beach in order for it to get this IRAC signoff, that meant we had adefinition of clean…that was useful…(BI09)
End-points suggested to evaluate this objective were often indicators of long-term impacts.Ultimately, of course, people are concerned about long-term effects. To gauge long-termimpacts indicators observable in the short-term were suggested:
• Its relative to the amount that’s spilled, and that’s a starting point…at best realistically,you’re doing a real good job if you get 20-25% recovered [on water] (BI10).
• A great deal of oil was recovered on shore…It was very costly but it was a very effectivebeach clean-up…Its very difficult to quantify the actual amount of oil on the beachbecause… mixed with sand and rocks and saturated in sorbent material… you can makewild broad estimates but its very difficult (BI10).
• No residual oil or buried oil that was going to show up later. If there was no residual oilor buried oil there, then that was the end of…our injury period (BI09).
The degree to which assessments about the way response was planned and implementeddepended to some extent on what is meant by “response”; a second objective in this category is
to integrate response, damage assessment, and restoration. Often the concern was expressedin terms of a desire for accurate information that could be used in damage assessment. Thiswas suggested by comments such as:
• Having that uncertainty as to what the IRAC sign-off date actually means, that kind ofcomplicates the matter. So if it were more closely linked to the actual cleanup date, thenthat would be awesome (BI09).
• One of the things we are trying to do is figure out how we can put restoration planningand response and marry them better so that we can make response decisions based onfuture restoration (BI08).
Third, an objective was that response should be started quickly. Local first responders shouldbe notified as soon as possible and crews and resources need to be deployed rapidly tomitigate impacts; time is critical. For example:
• I don’t think that the recovery of wildlife and the rehabilitation of wildlife has any bearingon the effectiveness of response? Oil is in the water and it will hit certain number ofcreatures before we are even around and that has no impact on response. There is animpact on how early you get your wildlife rehabilitation group involved; that’s were thecorrelation is – its not a correlation to the response itself…the direct relationship is howsoon have you called out experts and set up rehabilitation centers?...If you hesitate, youcan have a higher mortality rate and I have worked with a number of them that havegotten better and better at how they deal with injured wildlife…how fast can you get theMPs in? It has nothing to do with the response (BI10).
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which effective and timely response was achieved, shown in Table 10.
Table 10: Performance metrics related to effective and timely response identified ininterviews
Response performance metricsEnd-Point • Volume spilled compared to what’s removed from the shoreline
• What it looks like a year later• Evidence of oil on shoreline• How much oil was recovered off the water?• How much oil was recovered off the shore?• Endpoints in plan achieved?• Are desired endpoints achieved in reasonable amount of time?• Number of days until endpoints achieved?• Are response actions having the desired effect?• Members of the public voice support?• Number of public meetings organized?• Good working relationships with all parties involved?• command center and command personnel located in accessible location
Process • Cleanup of impacted areas organized to be manageable and able to monitor?• Access established for recovery and clean-up crews through private property?• Plans communicated day in advance?• Location of command center convenient and easily accessible?• Ability to effectively and timely monitor what happening in field?• Muster all forces in town?• Local officials set up task force for how to handle spill response?• Clear and reliable communication technology available (e.g., cell phones working)?• Do efforts correspond to tides?• Are there clear protocols?• Are there clear schedules?• Data sheets available and sufficient?• Are clean-up crews assigned effectively to do a good job?• Recovery or rehabilitation of wildlife conducted?• How soon have experts been called and set up triage and rehab centers?• Time it takes to implement tasks, such as boom deployment?• Is entire shoreline being inspected – both oiled and non-oiled?• How often is beach searched, all shoreline searched?• Is all pertinent data gathered and recorded?• Basic training for volunteers on bird collection and rehabilitation conducted?• Equipment and personnel demobilized when no longer needed?
Resource • Number of people in the field?• A ‘bird searcher’ on each SCAT?• Hours worked?• Supervisors out in field?• Number of monitors in field to give direction and warning to clean up crews• Resources adequate for planned tasks?• Amount of boom deployed?• How much money spent?• Types of skills represented on team?• Number of volunteers?• Number of packets of baby oil for oil removal distributed
Achieve legal and regulatory requirementsSome response goals are driven by regulatory and legal requirements. For example, oil spillresponse is governed by OPA ‘9010. Sec. 10002(a) states that “each responsible party for avessel…from which oil is discharged…is liable for the removal costs and damages…that resultfrom such incident”. It was often stated that another requirement is that sites affected by the spillmust also be returned to their conditions prior to the spill. Specific response and restoration
objectives can be expressed by end-points defined in response plans. As one intervieweenoted,• I was satisfied with the cleanup endpoints…since we had actual [interagency agreed upon]
definitions of what happened or had not happened on a beach in order for it to get this IRACsignoff, that meant we had a definition of clean (BI09).
Of course, this underscores the ways that response and restoration are related; responsestrategies can be influenced by the restoration goals that are desired.
Similarly, one interviewee noted that metrics for assessing response may not be the same asthose for natural resource damage assessment:
• The IRAC sign-off generally happened a lot later than the actual clean-up wasfinished…in damage assessment we are looking at the impact over time…we want toknow how for long might birds be exposed [to an oiled beach] or how long did thatconcentration stay there so that maybe it chronically poisoned someshellfish…having that firm data point that we have confidence in prevents negotiationand discussion from the RP later…if we can firmly say, “this is when it was cleaned”versus a sign-off date but it was cleaned a lot earlier than that that, it just providesuncertainty and needless discussion (BI09).
This underscores why some interviewees assess the objective of ‘gathering the right informationduring response’ on the basis of whether information relevant to damage assessment isgathered.
Meeting legal requirements also has links with an additional objective of gaining the support ofthe public (see below). One interviewee identified the important role of a Licensed SiteProfessional (LSP) as required by Massachusetts law:• It provides a more complete conclusion to the event; in any other spill situation it would
have been over two years ago and the LSP is still responding to complaints, reports of oil,etc. …The people of Massachusetts have a go to place for relief in the event that they findoil; that’s a good thing (BI10).
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which regulatory and legal requirements were met, as shown in Table 11. Metrics wereprimarily based on measurement of end-points, and not as much on process or resources.
Table 11: Performance metrics related to achieving legal regulatory requirementsidentified in interviews
• No oil should come off to the touch• Shoreline back to conditions prior to spill?• evaluate response with respect to endpoints achieved• For sandy beach no visible oil• For groin (jetties between properties), riprap no sheen or no oil available when touched• For marshes no sheen• No odor of oil
Process • Number of days until endpoints achieved.
Resource
Address needs and concerns of the affected public/communitiesInterviewees expressed the importance of understanding and considering the public’s concernsabout the spill and its response. Spill managers understood that local residents have a varietyof important concerns. For example, according to one official, members of the public want toknow most about how the spill will impact them in both the short and long-term:
• The public wanted to know what’s happening and how long will this cleanup take? Is itsafe for me to swim? Why aren’t you out here cleaning my beach...How and why did thishappen? What are you going to do to make sure it never happens again...They want toknow about wildlife, shellfish, are the lobster affected? How can I get my boatcleaned…My property is damaged because of cleanup contractors so when are yougoing to fix that…Will my well drinking water be affected? Are my children safe? What doI do if I get oil on me…We want local people to get jobs on these cleanup crews…whyaren’t our trained unemployed people working on this spill? What are you going to doabout? (BI02).
To address the needs and concerns of the public certain objectives must be met; theirperformance metrics are listed in Table 12.
First, provide accurate and timely information to the public. Interviewees identified this as animportant objective, even while some also noted the ways it could conflict with gaining supportfrom the public (see below):
• They wanted to know when it would be over. We actually could give it to them its justthey weren’t happy with it…They want it gone now (BI08).
Second, achieving this objective meant listening to their concerns, even if they could not beaddressed to their complete satisfaction:
• Public wanted assurances that there beaches would be cleaned up to the level of theirexpectation…Important to understand public’s issues; whether you agree with them ornot, they are their issues (BI10).
• If you try to set expectations that you are going to make the public happy, there is noway you are going to do that…No matter what you do or how good you are, how manyresource you bring to play, how quickly you get those resources out there, how great youthink you did in cleaning up the beach, no one is going to be happy. I can not tell youhow many of those meetings I walked out of thinking “wow, did I take a beating” (BI02).
Table 12: Performance metrics related to addressing public concerns and needsidentified in interviews
Response performance metricsEnd-Point • Level of staining and other aesthetic damage?
• Complaints about stained rocks?• Residual oil on shore (tar balls)?• Number of phone calls to complain?• Spirit of state regulations for public involvement was met
Process • A timeframe for ending the cleanup is established• Ongoing monitoring and addressing of issues post-spill?• People have a place or someone to go to with concerns?• Public receives assurances that beaches will be cleaned up to the level of their
expectation?• Information fairs and public meetings set-up?• A forum is provided to public so they can hear what’s going on in response action and
give their feedback• Number of public meetings• Level of conflict/anger• People are happy• Number of visuals for media• People given examples of what was impacted and what kinds of cleanup was going on• Public provided the kinds of information it wants?• Establish and keep up to date website for public information
Resource • Number of pamphlets distributed to inform public of hazards• Number of stakeholders involved in setting clean-up standards• Number of dispatch teams arranged to reach-out to various stakeholders• Frequency of information postings on BBP website• Number of flyers passed out about educating public on risks
Gain public support for the responseAn interesting objective that emerged from our interviews was the desire to gain public supportfor the response effort both during and after the response itself. The interviewees that raisedthis objective were mainly federal and state officials. This is one of the reasons that mitigatingsocial nuisance impacts was deemed important to some interviewees. This objective wasreflected in comments, for example, about the need to “manage” expectations and the wayslocal residents concerns could drive clean-up actions. For example we were told:
• Is reacting to that (zero oil tolerance) does that mean that’s a successful spill? Probablynot. You are not doing the environment any favors by cleaning to that level. You arecleaning because you have some irate homeowner or some irate landlord (BI08).
Within this broad category individuals identified several specific objectives.
First, manage media reporting so that it was accurate and supportive.• You can have a great response and do poorly in the media and it’s a bad response; and
you can have a poor response and do well in the media and it’s a greatresponse…Media is really concerned with numbers; how much is spilled, how much wascollected, how much is on shoreline, how many birds died, how many pounds ofplankton died? ((BI08).
Second, gain the trust and confidence of stakeholders. For example, we heard that part ofdoing the right thing is how one interacts with the different stakeholders:
• People can tolerate a lot more if they feel like they are being cared about (BI08).
Third, achieving this objective also meant listening to their concerns, even if they could not beaddressed to their complete satisfaction:
• As long as you provide a forum to the public to hear what’s going in the response actionsand at times give their feedback, then you’ve done your job. If you try to set expectationsthat you are going to make the public happy, there is no way you are going to dothat…this is not the kind of event that’s going to make anyone happy (BI02)
Fourth, develop constructive relationships (i.e., trust) with and among local officials:• Once we made good end roads with the Fire Chief…we had an asset in the community
and that made a huge difference (BI08).
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which response managers were able to gain the support of the public, as shown in Table 13.Metrics based on measurement of outcomes, or end-points, were identified in some cases.Many of them are the same as those identified for addressing public concerns (see above).
Table 13: Performance metrics related to gaining support of the public for responsedecisions and actions identified in interviews
Response performance metricsEnd-Point • Members of the public voice support?
• Level of staining and other aesthetic damage?• Complaints about stained rocks?• Residual oil on shore (tar balls)?• Number of phone calls to complain?• Was there trust from the public?• Public comments of support from critics and local residents (about response effort, not
about outcomes)?• Spirit of state regulations for public involvement was met
Process • A timeframe for ending the cleanup is established• Ongoing monitoring and addressing of issues post-spill?• People have a place or someone to go to with concerns?• Public receives assurances that beaches will be cleaned up to the level of their
expectation?• Information fairs and public meetings set-up?• Relationships and trust with local officials developed?• A forum is provided to public so they can hear what’s going on in response action and
give their feedback• Number of public meetings• Level of conflict/anger• People are happy• Number of visuals for media• People given examples of what was impacted and what kinds of cleanup was going on• Public provided the kinds of information it wants?• Quality of questions from media• Establish and keep up to date website for public information• Able to ‘stay on message’ during public meetings, press conferences, etc.?• Unified Command accessible for public questions and comments?
Resource • Number of pamphlets distributed to inform public of hazards• Number of stakeholders involved in setting clean-up standards• Number of dispatch teams arranged to reach-out to various stakeholders• Frequency of information postings on BBP website• Number of flyers passed out about educating public on risks
5.0 Objectives and performance metrics in the Chalk Point responseIn this section we describe our main findings related to the Chalk Point case study.First, through review of written documents and our coding of interviews, we identified from theinterviewee’s perspective, the important objectives of this spill’s response. We grouped theseobjectives into several categories.
Then, from the interviews, we identified the metrics that individuals used to assess the quality ofthe spill response by asking, “Was it a successful response or not? Was it a ‘good’ response ornot? On the basis of what are these judgments made?” Metrics, by their very definition, areused to assess progress toward specific goals or objectives. We provided, in tables, the metricsidentified by our interviewees associated with each objective. In some cases a metric wasassociated with multiple objectives.
Finally, we discussed what people told us about the quality of proposed metrics. In ourinterviews we asked that people explain to us what metrics were used (or ones they have seenused), ought to have been used and which ones they prefer. This provided them an opportunityto reflect on the quality of particular metrics and the difficulties of applying them in any sort ofassessment of the response. These challenges and limitations are discussed as well. We do notindicate for all metrics whether or not interviewees characterized them as “good” or “bad” – thisis being left to future project tasks, although we do provide some examples of what they said.
We found that many different objectives for the response to the oil spill were important to ourinterviewees. They can be grouped into the following broad categories:
• Protect worker and public health and safety• Protect environment and mitigate environmental impacts• Protect cultural resources• Meet legal and regulatory requirements• Mitigate economic impacts• Establish a coordinated and effective response framework• Implement an effective and timely response• Address needs and concerns of the affected public/communities• Gain support of public
Each of these is discussed in the following sections.
Protect public and worker health and safetyThe protection of worker and public health and safety is the first goal of the incident commandsystem.11 The priority of health and safety is reflected at all levels of government involved in theresponse. As one federal official put it:
• There is no life threatening situations to human health what so ever (CP02).
The interviewees expressed a concern particularly about public health in the Chalk Point spillresponse. A primary concern was the potential consumption of contaminated seafood:
• There were concerns come fishing season…they did put a moratorium out there for afew weeks…I don’t think they found… high levels of PAHs12 (CP05).
In regards to worker safety, concern was expressed in terms of the need for safety training ofclean-up crews:
• We didn’t do a good job of safety training…they probably, we should have been moremeticulous in that safety issue (CP01).
Interviewees identified very few performance metrics that could be used to assess the degree towhich health and safety were achieved, as shown in Table 14, in spite of its importance as agoal. This may be a reflection that counting injuries and fatalities is relatively easy and is themost direct performance metric.
Table 14: Performance metrics related to protecting health and safety identified ininterviews
ResponseEnd-Point • Presence of PHCs in water samples?
• Concentrations of oil in fish samples• Level of PHCs in clam/oyster tissue• Number life threatening situations to human health• Toxins in smoke plume if do in situ burning
Process •
Resource
Protect environment and mitigate environmental impactsMany interviewees spoke of the importance of protecting ecosystems. In some cases theirfocus was on habitats, such as wetlands, aquatic habitats, and marsh areas. Others expressedconcern about specific species, such as Ruddy Ducks, osprey, and fish (at both population andindividual scales). One interviewee spoke of his concern about river resources, beaches,marshes, benthic communities, and endangered species (CP04).
Within this broad category individuals identified several specific objectives.
First, individuals wanted to prevent and mitigate impacts to critical habitats or areas. Forexample:• Identify the sensitive areas where booming should take place (CP05).
Second, others identified the important of protecting a variety of species, including birds,oysters, fish, and terrapins. For example, benthos organisms were of particular concern to oneindividual because they are incapable of moving (CP04). Another was concerned aboutprotecting oysters because they are ecologically critical for the bay; oysters filter out algae fromalgae blooms (CP01).
Birds in particular were mentioned often as being of particular concern, and the objective beingto prevent or mitigate impacts to them. Concerns were stated about migratory waterfowl (CP01;CP07) and endangered species (e.g., bald eagles) issue. They were mainly concerned aboutthe overall population health rather than a focus on individual birds:
• We were observing productivity…particularly bald eagles (CP04).
Finally, a number of individuals identified the mitigation of ecological impacts from clean-upactivities as an important objective of spill response activities. They were concerned about
mitigating impacts from clean-up related activities such as in situ burning, as well as impacts tobeach areas, marshes, and birds:
• They [response crews] did a good job cleaning it up…everybody’s concern was that by being inthe marsh, all you’re doing is pushing it [oil] down further; you should just let it biodegrade(CP15).
• Can we dredge a canal or a channel through this one section…there’s a (historical) site heresomewhere; if we dig a channel will it affect it? (CP06).
Interestingly, a couple of people talked about the way that clean-up could exacerbate anexisting problem by focusing efforts to protect and manage problematic species, particularlymute swans:
• In the Chesapeake Bay we have a big issue with mute swans; we’re trying to eradicatethem…they cause problems with submerged-aquatic vegetation beds…the native swanpopulations get driven out because of consumption [competition] (CP05).
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which protection of ecological systems was achieved, as shown in Table 15. Metrics basedon measurement of outcomes, or end-points, were identified.
Table 15: Performance metrics related to protection of environment and mitigation ofenvironmental impacts identified in interviews
Response performance metricsEnd-Point • Number of oiled swans
• Number of fish killed• Damage from clean-up actions remediated?• Number of animals saved• “Appropriate” wildlife saved (e.g., not exotics or ‘varmints’)• Oiling of osprey eggs• Number of dead ducks• Number Injured birds around marsh area• Oiling on Diamondback terrapin eggs• Mortality of wildlife• How far sheen at surface extended out [miles]?• How long oil stayed?• Presence of odors of oil• How much grass was destroyed?• How many acres of marsh were impacted?• How many oiled birds were observed?• Dead and stressed organisms found (rather than estimated)?• How long it takes to reach background levels/concentrations• Observe water blowing over booms• How long does it take for area to recover from clean-up related damage?• Degree of change to beaches and sandbars from clean-up actions• Types of animals and vegetation present after spill cleanup• Areas that were impacted cleaned up to conditions before the spill happened
Process • Accurate cataloguing and enumeration of what found• monitoring stations established• Deploy booming – and double-booming in sensitive areas• Immediacy of rehabilitator organization’s response to call for assistance• Change of helicopter flight patterns in response to requests from biologists to
not disturb nesting birdsResource • Presence of booms
• Amount of oil containment boom deployed
Protect cultural resourcesFederal policy developed under the under the National Oil and Hazardous Substances PollutionContingency Plan (1998) specifies that states must have a contingency plan for archaeologicalunderwater historical sites in case of an oil spill. In the region affected by the Chalk Point spillseveral potential sites of historical significance were present. Ultimately, two received specialprotection during the response. The concern with such sites arises because of the effects of thespilled oil. However, one interviewee also elaborated that one of the biggest challenges toprotecting cultural artifacts, such as shipwrecks, is avoidance of damage from clean-up. Forexample,
• Its not so much the oil spill that does the damage…it’s the clean-up…if you burn and youhave wooden hulks sticking out of the water…or you get these chaps coming in to clean-up and say, “hey cool artifacts”…if we dredge a canal or a channel through this onesection…there’s a (historical) site here somewhere; if we dig a channel will it affect it?(CP06).
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which economic impacts were mitigated, as shown in Table 16. Metrics based onmeasurement of outcomes, or end-points, were identified. Here a concern was expressed notonly about end-points, but about effort and process. One metric suggested was that“appropriate” response actions be used. However, the definition of what is appropriate isinfluenced on the type of oil spilled. For example, if the oil floats, there is less need to worryabout it affecting, for example, nearby sunken ships (CP06). However, if the oil sinks thosesites may be of more concern. “Appropriateness” can also depend on the kind of artifacts orsites that are at risk. Chemical dispersants may not pose a risk to future research on the sitewhen its age is already known, but they may make it difficult or impossible to use carbon datingfor some sites (e.g., prehistoric sites). Similarly:
• They used fluidization where they super-saturate the ground to force the oil to thesurface and then clean it up…now archeologically, if you are looking at a stratified site, isthis going to ruin your stratigraphy? Is this going to force artifacts out of context so youend up with a layer with all your artifacts in it and you don’t have any stratigraphy(CP06).
Table 16: Performance metrics related to protection of cultural resources identified ininterviews
Response performance metricsEnd-Point • Number of critical sites protected
• Residual presence of oil on artifacts• Smell of oil• Soil concentrations of oil
Process • Did trench digging affect sites?• Experts contacted early for input about sites potentially at risk?• Command responsive to requests for protection of sites (e.g., deployment of
booms)• Were less destructive response actions chosen (e.g., sorbents and booms
rather than burning)Resource • Number of GIS & hard maps
• Number of laptops• Amount of boom deployed• Accurate spatial information about location of sites and artifacts
Mitigate economic impactsUnderlying many of the concerns expressed in the interviews was the objective of minimizingeconomic impacts from the spill through the response. Federal officials talked of their fiduciaryresponsibility to minimize costs associated with the response and damage resulting from thespill. As noted above, the most cost-effective restoration project is required by OPA ’90 law.However, it can be controversial at the time a decision must be made (rather than in hindsight)whether a response strategy, like burning, is fiscally responsible. For example, one intervieweestated that:
• In retrospect, if they actually lost those power lines and fixed them in a day or twoprobably would have been a lot cheaper than what they actually paid for in that spill(CP02).
Similarly, in regard to restoration efforts:• It [the oyster reef project] was a waste of money because the native oyster in
Cheapsepeake Bay is just about extinct [as a result of disease]…the money could havebeen spent better (CP12).
Within this broad category individuals identified several other specific objectives:• mitigate economic impacts from lost recreation,• reduce the economic impacts to local commercial fishermen,• mitigate economic impacts to local residents, and• mitigate economic impacts to towns from costs of clean-up.
Towns incur costs associated with local officials (e.g., harbor masters) working long, extra hoursduring a response. In addition, local equipment can be used. It was important to someinterviewees that Pepco made restitution, which they did, to counties for employee overtime andpersonal equipment used in helping response efforts (CP08).
Several interviewees spoke of way that public concerns can direct response by raising specificconcerns to address. These concerns may not be shared by officials involved with planning andcarrying out the response:
• You’re hearing from people what is important, particularly from the public perspective.And that certainly drove I think some of what we did with responding to those questionsthat we were getting…I wasn’t that concerned about fish but in order to satisfy the public,you just go out an collect fish to have data to show that (CP01).
Similarly, this interviewee spoke of the ways that scientists can influence costs by promoting“pet” research projects. It turns out that the Chalk Point spill was located nearby two researchcenters where some scientists were conducting research in the areas affected by the spill:
• A good thing in the sense that there was a lot of data that had been collected on the riverabout what natural resources were there and there was some historical data we couldcompare. It also meant there was a lot of pressure to do a lot of studies. We had someresearchers that wanted to do some work, get paid for it (CP01).
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which economic impacts were mitigated, as shown in Table 17. Metrics based onmeasurement of outcomes, or end-points, were identified. Interestingly, for this objective,measures that are often related to resources, such as total cost of clean-up, are here related toend-points.
Table 17: Performance metrics related to mitigating economic impacts identified ininterviews
Response performance metricsEnd-Point • Devaluation of property?
• Number dead fish• Number dead ducks & geese• Lost wages to watermen• Number of lost fishing days• Dollar value for days of beach closures• Change in earnings relative to previous years• Cost of response actions compared to value of lost resources (e.g., artifacts)• Duration of recreational fishing closures• Duration of beach closures• Presence of PHCs in water samples• Increase in crime rate [in southern MD due to influx of people from cleanup crews]• Monetary compensations• PEPCO made restitution to counties for employee overtime and personal equipment
used in helping response efforts
Process • Funds spent on important and appropriate projects• RP cleaned boats that were soiled with oil from spill
Resource • Amount of money spent on studiesTotal cost of clean-up
Establish a coordinated and effective response frameworkMaking the best possible decisions with limited information, uncertainties, and time pressureswas identified as an important objective by many interviewees. While they found it difficult toarticulate specific measurable performance metrics for them, it is nevertheless important as theyview good decision making to be closely associated with good outcomes. Within this broadcategory individuals identified several specific objectives.
First, coordination among participating government agencies and contractors – the responders– is important. Coordination affects the quality of plans and decisions (i.e., obtaining the rightinformation in a timely manner) and how well plans are implemented. Good coordination wastightly coupled with concerns about good communication and the ability of federal responders towork together in a “team” atmosphere, including building trust and good relationships. Forexample:
• By the time the first 48 hours (48-60 hours) were over with…I think they really got theiract together and people were trying to coordinate with each other as best they could(CP08).
• When confronted with a dual problem like that its far more important to fix what’s goingon in the command post than to fix what’s going on in the field because eventually youare going to get all that oil back (CP09).
Second, strong and experienced leadership is viewed as important. Some intervieweesassociated good decision making with experience and the ability to generate respect,particularly from local knowledge:
• The contractors were put in the position of leadership very early on and I think this wasbecause they had experience; they quickly fell into leadership roles by default becauseleadership from agencies was weak. I thought this was quirky and undesirable becausein the end they were working for their client (RP) and not the public where these publicagencies were accepting a leadership role in the response from a representative of a
group responsible for the spill that does not have the public interest as its first stake,they have their own interest as their first stake and that just should not have been; theresponse should have been in the interest of the public who is the damaged party(CP04).
• …a brand-new rookie OSC and no experience as an OSC and no emergency responsebackground (CP09).
• probably never seen good leadership; so she was thrown into this task that requiresmilitary kind of leadership, with large amounts of troops and huge amounts of equipmentand so she was unprepared. Not her fault just that the EPA system and CG system arevery different in how they respond to spills and the way they teach management andleadership (CP02).
• I was frustrated that we were receiving information from somewhere out there that wascoming into EPA that they were tracking about the spread of the spill and the extent ofdamage and I knew from putting people on the beach from my own communicationsystem that there data was poor… When we brought this information back to HQ,because of the chaotic situation, they were not in a situation to look at us and say howdo you have this information…and then track it (CP04).
Others related it to staff stability. For example, one state official told us that:• the rotation of NOAA personnel in and out of the site was really astonishing…we would
have a rapport with someone, figuring out what they knew, what they did and then wouldgive us an idea of what was expected of us…and then 48 hours be gone replaced bysomeone new (CP04).
Third, and closely related to strong and experienced leadership, is the objective of creating awell-organized command, with a clear chain of command:
• Initial response was not so successful and this is the most critical time period. If youhave success there you can chalk it up to being a successful event- it was toodisorganized, no chain of command (CP11).
• Spill was like the fall of Saigon with cell phones (CP03).• Not sure that the EPA had a good understanding of what Incident Management was at
the time; they were in various stages of disarray of how they were going to do things andwho was running what (CP08).
And, fourth, of establishing clear and achievable goals:• Because there was not much of a Unified Command, they were not able to decide as
group that they were done (reached an endpoint); so they have gone through a constantprocess of monitoring, reviewing the case over and over again, still not able to saythey’re done cause the standard can’t be met; Once the standard is set, it can not bechanged (CP02).
• The endpoints we established were not realistic…clean till there is no oil (CP16).
Fifth, it was important to gather accurate and relevant information for decision-making.Moreover, it was not only important to have accurate and relevant information, it was importantto have it quickly. For example:
• IC would argue that they did not have any reports of the facts of what he was tellingthem of where oil was… they (IC) did not know cause they were not out there so howcould they argue with us that its not there (CP04).
• A colleague…called me and said, “I don’t know what they told the NRC but this is a lotbigger than what they really said and you better get down here” (CP09).
The ability to generate such information is dependent, in part, on the availability of appropriateequipment, such as computers and GIS software to support analysis of spill and responsestrategies.
Sixth, an effective response requires that a well-coordinated scientific effort drive the gatheringof decision-relevant information. Expert knowledge should be used, and it should be transparentthat such information was used. It was problematic when people felt that decisions were drivenby “inappropriate” information:
• Don’t let public concerns and perceptions drive response actions and priorities (CP06).• Difficulty in convincing people (public) that everything was fine after sampling tests
proved so (CP03).
Concerns about this are discussed further below in the section about mitigating economicimpacts (public pressures for response actions and researcher pressures for studies).
Seventh, it was also important to create a flexible and learning organization that couldincorporate lessons from past spills and responses and develop lessons for future responseefforts:
• Companies that have had a lot of big spills…when they have a report of a spill, theyassume its catastrophic and throw all of their resources at it and then if it turns out to beno big deal, they send people home. Pepco did not do that and they really missed anopportunity to but some good will for themselves on the part of the locals as well as theresponders (CP09).
• We didn’t have anything to respond with, we still don’t… none of the 3 counties…haveany oil spill equipment (CP08).
• We did numerous post-mortems on it…and there were a few issues still but mostly it waspeople would have liked to have been involved earlier (CP06).
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which economic impacts were mitigated, as shown in Table 18. Metrics based onmeasurement of outcomes, or end-points, were identified. As might be expected, many of theproposed metrics have to do with process, as one way of measuring “good” decision-making isby the way that decisions are made. It can be difficult to assess the quality of a decision by theoutcome because there can be many intervening factors that affect outcome.
Table 18: Performance metrics related to establishing a coordinated and effectiveresponse framework identified in interviews
Response performance metricsEnd-Point • Clear chain of command established?
• Incident Command System used?
Process • Number of hours to set-up incident command center• Number of daily meetings• Daily reports for morning meetings held by UC?• Informed of meetings in advance (i.e., lead time)• RP is responsive• Key people became involved early?• Is Incident Commander able to reach key people, does their phone number work, did
they respond?• Clear communication protocols?• Stable command posts established?• Time taken to re-staffing response people after contractor fired• Chauvinistic behavior among staff in command center?• Frequency of resource and personnel changes• All federal responders notified quickly?• Pick-up and shipping schedule for waste generated by clean-up organized?• Response organized by discrete (river) segments• Frequency of meetings• Accurate information obtained from the wildlife surveys and SCAT teams• Clear standards for sign-off established• People in field used to verify accuracy of information• Did Unified Command resist information that did not conform to their expectations?)• Arguments and lack of cooperation among Unified Command• Clarity to all parties about stages of response effort• Clear understanding of rights the state trustees have as a state agency
Resource • Number of supervisors assigned per section• Enough Spanish speaking crew supervisors?• Number of radios available• Availability of GIS and computers
Implement an effective and timely responseA critical objective the interviewees expressed was that response should be effective and timely.In other words, the response should achieve the established goals and do so quickly in order tominimize impacts from the spilled oil. While this set of objectives is very outcome oriented,metrics to assess them are also related to process and resources.
Within this broad category individuals identified several specific objectives.
First, recover and remove as much oil as possible:• The best things would have been to contain it…we tried to do protective booming
(CP01).• We have since debated in situ burning…and the state representatives all agree that it’s a
good strategy and it will never happen…you just don’t recover a lot of oil withmechanical [equipment]…in situ burn can burn up to 95% of it (CP03).
But there are consequences:• If we can sacrifice a short-term air release vs. long-term stuff that we’re doing here, its
well worth it…but the circumstances need to be just right (CP03).
Second, ensure timely and effective implementation of response actions. For example,response actions should be quick (CP01; CP11). One positive example given was theimmediacy of the rehabilitator organization’s (Tri-state) response to call for assistance:
• We called Tri-State and they were here in an hour and a half on site (CP05).
Third, an effective and reliable monitoring system should be established. This is related to gooddecision making, but here the emphasis is on the effectiveness of plans and theirimplementation. For example:
• We told the RP several times to boom off [certain area] but they did not do it…we shouldhave monitored it to make sure it got done (CP16).
• They need to re-think what they need in a monitoring system, checks and balances(CP15).
Fourth, the degree to which assessments about the way response was planned andimplemented depended to some extent on what is meant by “response”; an objective in thiscategory is to integrate response, damage assessment, and restoration. Often the concern wasexpressed in terms of a desire for accurate information that could be used in damageassessment:
• The NRDA process will never go smooth …because of the uncertainties (CP12).• There is usually minimal knowledge about damage assessment or restoration cause you
don’t go to school to learn that stuff, there’s no way you can (CP07).
Fifth, an important factor in achieving goals is the establishment of clear protocols that allowpeople to understand how to make decisions and to be consistent. For example:
• Sheen is a good measure for me because you can send people out to do it and youdon’t need any tools or equipment or training…sometimes the very crude methods arebest (CP04).
• Trying to get everybody to agree on what’s acceptable…I think all the people that havethe authority to sign-off should be doing the sign-off at the same time…it might not bethe same group going out the next time (CP05).
On the other hand, others think that clear protocols may not necessarily translate to reasonableendpoints; thus, one interviewee suggested an awareness of how they make trade-offs:
• The law says no sheen but the fact is that’s a standard that can’t be met in fact if you tryto get a marsh to no sheen standard, you destroy the marsh; you try to bring practicalityto, often, angry people (CP02).
Finally, an objective expressed by interviewees was the need to apply adequate resources toget the job done. Resources include equipment, such as booms and computers, and people,such as staff and clean-up crews. Moreover, it is not enough to just have such resourcesavailable – they must also be used effectively and efficiently (CP10; CP06; CP04; CP03).
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which economic impacts were mitigated, as shown in Table 19.
Table 19: Performance metrics related to effectiveness and timeliness of responseidentified in interviews
Response performance metricsEnd-Point • Number feet of shoreline impacted
• Presence/absence of sheen• Number gallons oil recovered• Number of days company (PEPCO) was shut down• No oil in water• How long takes to reach endpoints• Oil present in sediments?• Areas that were impacted to be cleaned up to the ecological state the
environment was before the spill happened• Patchiness of oil• Solid sheet of heavy oil• Presence/absence of tarballs• Is oil being contained?• Number of areas to be signed-off compared to Number actually signed-off• Number of areas that were identified as sensitive areas ordered to be boomed
off but order was not followed through and area impactedProcess • Digging holes to look for oil on shoreline?
• Quarterly checks to see if oil is present or not on beaches• Breach of water over boom?• Protection equipment put in the right place at the right time?• Number of times a single order was given but still not done• Accurate reporting and counting of crews in field• Number of newsletters published by RP• Number of public meetings held by RP• Time it takes to get response in order• Number/frequency of fly-overs for real time aerial photography• Time taken to set-up boom• Clear chain of command established?• Lead resources mobilized?• Communication funneled out to the appropriate people?• Quality of contractor work (e.g., perform required tasks?• How quickly SCATs out in field?• Time it takes to resolve bad cell phone coverage, radio issues?
Resource • Cost of response• Number of cleanup teams and people on crews• Amount of money spent by RP• Number of laptops• Number of GIS & hard maps• Amount of equipment brought in • Pounds of sorbent material used• Amount of oil containment boom deployed• Number of people working at one time• Number of people per SCAT
Meet legal and regulatory requirementsSome response goals are driven by regulatory and legal requirements. For example, oil spillresponse is governed by OPA ‘9013 Sec. 10002(a) which states that “each responsible party fora vessel…from which oil is discharged…is liable for the removal costs and damages…thatresult from such incident”. However, according to this same Act, the most cost-effectiverestoration project should be chosen as required by law (CP07). It is possible for someone to filea FOIA to verify if this was in fact the case. The reason for this was identified as:
• We have to present the RP with a reasonable list of things we want to do…we can’t hitthem for the highest ticket price (CP07).
As described in a previous section, the National Oil and Hazardous Substances PollutionContingency Plan (1998) specifies that states must have a contingency plan for archaeologicalunderwater historical sites in case of an oil spill. This underscores the ways that response andrestoration are related; response strategies can be influenced by the restoration goals that aredesired.
Such legal requirements can affect certain choices made during a response. For example:• We can’t do certain things without approval…we might be able to burn it, but everybody
needs to be on board (CP02).
At the same time, there is awareness expressed by some of the interviewees that followingregulations does not necessarily mean that a response was “good”:
• NIIMS is not the end; it’s the means to the end…just because all the charts are filled outdoes not necessarily mean that we did a good job (CP03).
Only one specific performance metric for measuring the degree to which legal and regulatoryrequirements were met was defined by two of the interviewees, in spite of it being raised as animportant objective in multiple interviews (Table 20).
Table 20: Performance metrics related to meeting legal and regulatory requirementsidentified in interviews
Response performance metricsEnd-PointProcess • All procedures followed (e.g., NIMS)
Resource
Address needs and concerns of the affected public/communitiesInterviewees expressed the importance of understanding and considering the public’s concernsabout the spill and its response. Spill managers understood that local residents have a varietyof important concerns. Our interviewees acknowledged the importance of responding to thepublic’s concerns, addressing needs and concerns of the affected public/communities (but seeabove about not having decisions be driven by them), communicating well with public in timelymanner, establishing good communication with the public, and being sensitive to thecommunity. For example:
• [There were] various levels of confidence and trust that people had…public expectationsare very, very high and the reality of the situation needs to be presented to everybody(CP14).
• Could be an endangered species in a certain area so a specific cleanup method must beimplemented to not disturb animal…the public is not seeing the whole process (CP07).
When asked how one interviewee thought the communication process went with the public hecommented:
• All this was, was a venting, a lynching…no information was exchanged and this set thetone for every meeting after that… We had lots of very smart people (scientists) whodidn’t know how to talk to the public… they’re not telling the public what they want tohear… it doesn’t matter what they said because the public doesn’t think that (CP03).
In terms of restoration projects:• You have to find projects in the area because of the perceptions…people won’t have it
otherwise (CP07).• its important as trustees that we let the public know what’s going on in terms of
restoration (CP04).
Again, in the context of addressing concerns of affected communities the issue of whether“inappropriate” response actions were being driven by those concerns was very salient toresponders. For example, while one of those we interviewed felt that protection of oysters wasimportant for ecological reasons (see above), another believes that even though economicreasons were also suggested, “commercially [oysters are] just a relic industry but it is till part ofthe culture of the bay.”
Performance metrics elicited during the interviews are shown in Table 21.
Table 21: Performance metrics related to addressing public concerns and needsidentified in interviews
Response performance metricsEnd-Point • Public reimbursements for private property losses
• Number calls from public• Number of fliers and informational packets delivered door to door• Amount of oil removed manually from shoreline
Process • Immediacy of public meetings (number days after spill occurs)• Incorrect information disclosed?
Resource • Number hours agencies spent on public outreach (meetings)
Gain public support for the responseThis objective also emerged from the Chalk Point case study. The interviewees that raised thisobjective were federal and state officials. They spoke of managing the public:
• Maybe we could have gotten the word out better to the public so the public responsecould’ve been managed better than it was… maybe we could’ve helped the agencieswith their public meetings (CP04).
And persuasion – even while talking at the same time about cooperation and consensus:• You need to bring “angry stakeholders” together and try to reach consensus among
them in a cooperative kind of way that you can’t go into the marsh…,[because thedamage human cleanup activity will cause] and try and persuade them to come to that
conclusion…then arrive on an agreement of an endpoint that you can actually reach(CP02).
Within this broad category individuals identified several specific objectives.
First, it is important to show local residents that their concerns are being listened to andaddressed (see above).
Second, it is important to develop and maintain trust with members of the public.• Public loses confidence in information reported; you screw the public one time even if its
innocent…they will not believe a word you say (CP03).• People felt like PEPCO wasn’t telling the truth and not getting the information out,
whatever that information was (CP08).
Third, efforts to communicate with and engage with the community need to be proactive andtimely:
• Officials held the first meeting 5 days after the spill; it should have happened the 2nd day(CP03).
Fourth, it is vital to establish good communication with the public:• Provide public accurate information (CP03)
Interviewees identified a variety of performance metrics that could be used to assess the degreeto which economic impacts were mitigated, as shown in Table 22.
Table 22: Performance metrics related to gaining support of the public for responsedecisions and actions identified in interviews
Response performance metricsEnd-Point • Public reimbursements for private property losses
• Number calls from public• Number of fliers and informational packets delivered door to door
Process • Immediacy of public meetings (number days after spill occurs)• Incorrect information disclosed?
Resource • Number hours agencies spent on public outreach (meetings)
6.0 DiscussionIn this section we provide a discussion of important themes that emerge from the analysis andthe ways that objectives and performance metrics varied between the two spills and betweenthe different types of stakeholders (e.g., federal officials vs. local officials).
These need to be interpreted with caution and with an understanding of the goals of the casestudies. These case studies were designed to elicit a broad range of information about theobjectives desired among those involved with each of the spill responses and the ways that theyassessed the spill responses -- or thought that the response efforts should have been assessed.In other words, we asked for both a description of what happened and what they thought shouldhave happened and what metrics were used and should have been used. We did not designthis research to systematically compare the objectives and metrics that emerged from theseinterviews; this is a future task in this project. Thus, the following comparisons are suggestiveand preliminary.
We begin with a discussion of the kinds of spill response objectives expressed by those weinterviewed. We then turn to a discussion about performance metrics. We end with adiscussion about ways that people we interviewed for both of the case studies think aboutmetrics: what are the characteristics of good performance metrics and what are challenges tousing performance metrics for assessing marine oil spill response.
6.1 ObjectivesSeveral objectives for oil spill response emerged from each case study. And, many of the sameobjectives were identified among the respondents in both cases. They follow directly from thegoals of oil spill response as expressed in federal policy (see above, fromhttp://response.restoration.noaa.gov/book_shelf/910_response.pdf, pg. 9).
Mitigation of ecological impacts and economic impacts featured prominently as broad objectivesamong those we interviewed, as did protection of public and worker health and safety. Inaddition, goals associated with the ideas “stabilizing the situation,” “coordination,” and “timelyand effective response” are supported by the broad objectives of:
• establishing a coordinated and effective response framework,• implementing an effective and timely response,• meeting legal and regulatory requirements,• addressing needs and concerns of the affected public/communities, and• gaining the support of the public.
Maintaining safety and minimizing adverse environmental and socio-economic impacts alsodepend on meeting these objectives.
In addition to similarities on the broader categories of objectives, many of the more specificobjectives are shared. For example, interviewees in both cases wanted to protect sensitivehabitats and populations of threatened and endangered species. They were concerned withmitigating impacts from clean-up actions. They want to mitigate economic impacts related tolost recreation, tourism, fisheries, and to towns from their efforts during the response.Furthermore, they were concerned about the timely gathering and use of relevant, accurate, andcredible information for decision making. Strong and flexible leadership that can learn from pastexperiences was important to them so that the response could be well-planned andimplemented. And, they shared objectives related to addressing public concerns (e.g.,providing accurate information to the public).
At the same time, there are a few interesting differences, which are reflective of the particularcontexts of each spill.
First, mitigating impacts to cultural resources was identified in the Chalk Point case and not theBouchard-120 case. This reflects the presence of significant artifacts in the region affected bythe Chalk Point spill. It may also be a reflection of whom we interviewed – or more accurately,did not interview in the Bouchard-120 case. However, it is noteworthy that no one in theBouchard-120 case made mention of historical or other culturally significant artifacts.
Second, the mitigation of social nuisance impacts was not a category of objectives that emergedfrom our analysis of the Chalk Point interviews. However, some factors that can be related tosocial nuisance impacts were raised in the context of other objectives. For example, addressingthe potential for an “increase in crime rate [in southern MD due to influx of people from cleanupcrews]” (CP02) might be considered by some to be a social nuisance, but the interviewee wasclearly talking about this in the context of economic impacts.
Third, while people in both cases expressed objectives (and performance metrics) related toestablishing a coordinated and effective response framework, there were some differences inemphasis. These differences suggest the importance of the particulars of experience thatinform people’s views. Critical comments emerged about the integration of non-federal officialsand responders into the response effort. In the Bouchard-120 case this concern was raisedabout local officials and local first responders, but not in regard to state-level responders. In theChalk Point case we found the opposite.
In the Bouchard-120 interviews the roles and participation of local officials and local residentswas a concern. In fact, the way in which local responders along Buzzards Bay were contactedand integrated into the response was a source of contention and criticism among those weinterviewed. Location of the unified command was an example of ways that local officialsthought they were not included to the extent that they thought desirable. In addition, there wasa concern with how local volunteers were brought into the response effort in the Bouchard case.
On the other hand, among the Chalk Point interviewees neither of these two concerns werestrongly articulated; volunteers were not even mentioned in these interviews. Instead, therewas a concern with whether expert input (e.g., about cultural resources) was obtained and howwell federal and state agencies’ were coordinated (e.g., clarity about jurisdictions). In the ChalkPoint case a federal official spoke of the importance of involving local stakeholders in decisionsabout clean-up goals, but the metrics suggested were about one-way communication andoutreach to the public (e.g., number of meetings). There was also a concern expressed fromstate officials about their involvement in the response effort. In the Chalk Point case there wasmuch said about the quality of leadership (FOSC) and the coordination and communicationamong responding federal and state agencies. However, there are some indications thatinvolvement of local officials would have been useful in Chalk Point in terms of understandinglocal nuances:
• People don’t understand how strong that current is…it has one hell of a flow (CP08).
Another difference among objectives relates to the kind of people that expressed a concern withgaining public support for the response. Only state and federal officials spoke of this as anobjective. This is a very instrumental perspective that one often finds from officials involved withhazard management.
Finally, it is apparent that objectives can be in tension. Responders can have a difficult timebalancing the ways that they strive to achieve goals and objectives. This was expressed clearlyby a federal official involved with the Bouchard-120 response. Using a “best response”14
framework helps the Unified Command simultaneously focus on all the critical elements so thatnone are neglected: “As part of the best response we don’t get stuck on one area or issue”(BI11).
One interviewee stated that directing oil to “sacrificial areas” is more important to do well thanpreventing oiling anywhere:
• Direction I think is more important than stopping. If you can direct it into a “sacrificialarea” send it into a cove that yes will have an environmental impact but you will save thisamount of other shoreline. I think there’s an indication of a more approachable loop thanhaving this visual image of “I’m going to stop it all.” (BI05)
However, trade-offs can be made differently. For example, another interviewee privilegedmarshes and other sensitive areas saying:
• Sandy recreational beaches, it’s hard to compare say to a salt marsh because itdepends what perspective you are coming from, from an environmentalist perspective…I would say the salt marsh is much more important to protect… its also important toprotect those sandy recreational beach areas in part because that’s what connectspeople to the resource and imbues them with more of a stewardship ethic and it ties intothe public trust doctrine we all commonly own the ocean’s resources…at least our publicbeaches and I think its really important to maintain that connection (BI07)
As another example, an interviewee from the Bouchard-120 case expressed the objective ofminimizing the costs to the responsible party from the response: “be fair to the responsibleparty. Don’t over-burden responsible party.”
Another person spoke of this being an objective that might conflict with other important goals.When gallons spilled is used to determine size of initial response, it can result in too slow of aresponse:
• Because they were unsure about how many gallons had been spilled, the spin-up gettingthe clean-up contractors there on-scene started out pretty slow. Again, the CG is prettyresponsive to the RP and they don’t want to mobilize an entire army if it turns out thatthe spill isn’t as big then you’ve spent all this money when you didn’t have to…[later] theCG admitted that, that the response spin-up time was actually slower than it should havebeen (BI09).
While these examples show how there can be tensions among the objectives, we also foundthat objectives can be differently weighted. Based on our interviews it appears that somepeople believe there is a “pecking order” of objectives. Protection of historical underwater sitesmay not be the most important – there is need for trade-offs and making choices aboutpriorities:
• The whole point of a contingency plan is to prioritize; if they say, “ok, we need a pointfrom which to deploy boom and this one is most logical and there’s a site there, is itsufficiently significant to ask them to move or can you let that one go” (CP06).
14 Hereth, L. & J. Kuchin, Measuring response: A balanced response scorecard for evaluating success, USCG.Available at http://www.uscg.mil/hq/gm/mor/articles/proceed.pdf
One arena in which the tension among objectives is most readily apparent relates to objectivesof mitigating costs of the response, making informed decisions based on relevant information,and addressing public concerns and need. In some cases, such concerns were dealt with asrequested, but in others not:
• we had a rule of thumb that in rocky areas we would be better off letting some naturalcleaning to go on, so we were washing rocks but not high powered steam cleaning – butour rule was to let some natural processing go in certain areas. Well, there is rock in abay and the community said that is the swim test rock. We were not going to cleanit…all our children who learn how to swim have to swim to that rock…oh so you needthat rock cleaned and we will do it. [Another example is that]…we were trying tocommunicate with some communities that natural processing is the best alternative, andthis man wrote that I am 75 and I don’t have a lot of years left to watch naturalprocessing and I waited all my life to buy this waterfront property – I did the best I couldto listen to every person. We did not do anything to clean up this man’s property anybetter. We just cleaned it up [as we did other locations]. We just cant clean upsomeone’s beach better than others.” (BI11)
• The rumors were that there were competing interests for these cleanup crews…to somedegree CG wants to be responsive to the complaints of the public… clean-up crewswere so concerned to not damaging the habitat on Ram island… that when they firstsent their crews in they were hand-scrubbing the rocks…we finally told them that itsmore important that you get it clean cause they were afraid [of damaging the ecosystem](BI09).
These objectives identified as important during the interviews are all related to the protectionand promotion of what people value, such as protection of critical habitats and promotion ofdecisions based on the best information available. Why they value certain things can differ; theymay value certain outcomes for intrinsic reasons (e.g., value of species for their own sake) orinstrumental reasons (e.g., they allow other things to be accomplished) and the relative weightsgiven to each may also differ.
6.2 Performance metricsIn this section we further identify some of the similarities and compare differences in findingsfrom the two case studies, but our focus is on the performance metrics.
Overall, a large number of performance metrics were suggested by the interviewees. Theyrepresent a range of variables that can be measured quantitatively or qualitatively, and they canbe grouped according to end-point, process, and resource related metrics. While a largenumber of performance metrics were identified or suggested, a number of interviewees alsopointed out the shortcomings of some of them. For example, one interviewee noted that thenumber of volunteers taking part in the response may not be a good measure because there areconstraints on using volunteers (e.g., OSHA regulations) (BI07). She thought that a better metricmight be the number of people calling to volunteer. Further considerations of strengths andweaknesses of alternative metrics are discussed in sections below.
However, performance metrics are best defined with respect to specific objectives. This is whywe have organized the elicited performance metrics with respect to each objective. By codingperformance metrics in the context of particular objectives it is apparent that many metrics werediscussed for some objectives (e.g., mitigating ecological impacts) and few in relation to otherobjectives (e.g., protection of public and worker health and safety). This should not be taken tomean that those objectives associated with more performance metrics are more important thanthose with fewer performance metrics; this conclusion is not justified based on the evidence
available. One reason for this imbalance may be the perceived straightforwardness ofassessing some objectives versus others. For example, protecting health and safety can beassessed by counting OSHA-reportable injuries and fatalities and work-hours without accidents.Another reason may be an artifact of how we grouped objectives into larger categories. Moreobjectives are contained within the category of mitigating ecological impacts than of mitigatingimpacts to cultural resources or meeting regulatory and legal requirements.
Another issue is that a metric suggested in the context of one objective, may not have beensuggested again by the same interviewee in the context of another objective – even though theperson might believe that it would be appropriate. For example: mitigating social nuisanceimpacts and addressing needs and concerns of the public. Furthermore, one might argue thatmany of the end-point metrics suggested for assessing the prevention and mitigation ofecological impacts and economic impacts could also be measures for whether or not publicconcerns were addressed. However, they were not always expressed as such. Instead,addressing public concerns was often measured in terms of what effort was made for publicoutreach and involvement: How many meetings? How many leaflets distributed? Our inabilityto systematically identify all performance metrics with respect to each objective is a shortcomingof using interviews. This shortcoming is to be addressed by further research; we are developingmethods that will ask respondents to rank-order objectives and to rank-order performancemetrics for each objective.
When we consider the kinds of metrics elicited in the two case studies, one observation is thatthe performance metrics suggested for each case differed for some objectives. For example,making the best possible decisions with limited information, uncertainties, and time pressureswas identified as an important objective by many people we interviewed for the Chalk Pointcase. While this category of objectives was also raised in the Bouchard-120 interviews, itseems to have generated much more attention from the Chalk Point interviewees; this may be areflection of their perceptions of inadequacies during the Chalk Point spill response. While theyfound it difficult to articulate specific measurable performance metrics for them, it isnevertheless important as they view clear chain of command, strong leadership, clarity aboutresponsibilities and organizational jurisdictions to be closely associated with good outcomes.
Third, for almost all objectives performance metrics related to resources, processes, and end-points were suggested. Distinction among the three types of metrics proves useful forcharacterizing the performance metrics elicited by the interviews. However, in some casesmetrics of only one or two of these types were elicited during the interviews (e.g., protection ofpublic and worker health and safety, meeting legal and regulatory requirements).
Some interviewees argued, however, that it is not always useful to define performance in termsof end-points because they are difficult to measure in the short-term. For example, one personstated that the point was whether a comprehensive effort was made to recover all the wildlife(CP03). On the other hand, performance metrics related to mitigating economic impacts werealmost entirely related to end-points, rather than process or resource related. This may be areflection of the belief that economic costs are easier to measure than impacts to ecologicalimpacts. In fact, estimating economic impacts can be very difficult.
Similarly, as might be expected, many of the proposed metrics have to do with process, as oneway of measuring “good” decision-making is by the way that decisions are made. It can bedifficult to assess the quality of a decision by the outcome because there can be manyintervening factors that affect outcome.
Fourth, the end-points measured are rarely directly about, for example, the health or quality ofthe ecological system or its components, public satisfaction, public support, or economicimpacts. In some cases the metrics assess indirect or interim indicators of the state of anecological system, habitat, or population. For example, impacts to ruddy docks were based onextrapolations of empirical field data:
• There’s an extrapolation figuring “ok, if we found x number of ducks, how many wereactually killed... What happened to the reproduction success? Did that fail or reduce?(CP07).
Similarly, some performance metrics are direct measure of economic impacts, such as losttourism dollars, lost fishermen income, while others are indirect measures: length of shellfishbed closures, acres of shelffish bed closures, number dead ducks and geese, duration of beachclosures, presence of PHCs in water samples.
This was observed for assessing progress on objectives associated with particular substantiveoutcomes, such as economic costs, ecological health, and protection of cultural artifacts. It isalso apparent with objectives that are associated with the conduct of response effort:establishing a well-coordinated response, meeting legal requirements, and implementing aneffective and timely response. In these cases much of the attention is on measuring the qualityof the effort. End-points are related to effectiveness, but these cannot be measured directly. Forexample, the “quality of decisions” is difficult to assess on the basis of decision outcomes.However, how a decision was made or information gathered and validated can be moreappropriate.
Furthermore, end-points suggested to evaluate the effectiveness of response were oftenindicators of long-term impacts. A difficulty reflected by many of the suggested metrics is thatmeasures of effectiveness or ultimate impacts can rarely be made at the time of response.Eventual outcomes cannot be measured because the systems are dynamic, although measuresof interim conditions can be. This is a challenge to metrics assessing progress toward goalsthat take a long time to realize (National Research Council 2000a; see below). In the case of oilspills, a desire to assess the effectiveness of the response is really about wanting to know aboutlong-term conditions and how they differ from pre-spill conditions:
• The problem with trying to measure effectiveness is what are you measuring itagainst...the circumstances are different [in each spill]… Part of the measurement canbe restoration costs although again, that’s a measure of the impact as well (BI08).
Ultimately, of course, people are concerned about long-term effects. To gauge long-termimpacts indirect indicators observable in the short-term were suggested:
• Its relative to the amount that’s spilled, and that’s a starting point…at best realistically,you’re doing a real good job if you get 20-25% recovered [on water] (BI10).
• A great deal of oil was recovered on shore…It was very costly but it was a very effectivebeach clean-up…Its very difficult to quantify the actual amount of oil on the beachbecause… mixed with sand and rocks and saturated in sorbent material… you can makewild broad estimates but its very difficult (BI10).
• No residual oil or buried oil that was going to show up later. If there was no residual oilor buried oil there, then that was the end of…our injury period (BI09).
Finally, many of the suggested performance metrics are based on what can be counted orobserved (e.g., did it occur or not occur). Metrics that are easy to measure may be moreappealing from a bureaucratic perspective. Assessments of response effectiveness can bedefined more readily with respect to whether clear procedures and protocols were established
and followed. Similarly, how many gallons of oil and contaminated debris were removed? Howmany leaflets were distributed to local residents? How many birds were found dead or oiled?Were endpoints defined in plans achieved? Were state and federal standards for contaminantsmet?
• I have not been involved in a major oil spill where that [# of gallons oil spilled] has notbecome a hot potato…its an important metric because the amount of oil spilled willimpact the amount of resources (personnel and materials) that you call out… It’s animportant element and a first step (B10).
Of course, it might not be so easy to count such things accurately, claimed some of theinterviewees. For example, one person suggested that a metric could be an answer to thequestion, “are desired endpoints achieved in a reasonable amount of time:
• The number of days between initial oiling to final IRAC standards, sign-off. (BI09).
However, there is a difficulty with this measure; they are subjective and IRAC signoffs can comemuch later than the actual clean-up finished. This means that an objective measure with clearmeaning may be difficult to define:
• It [signoff] identifies the termination; it doesn’t necessarily say we are successful, itjust says we’re done; It doesn’t say we did the right thing, it just says we’re done;that’s your ticket home (BI08).
In fact, the focus was not always on quantitative metrics that many assume can be accuratelycounted. For example, a state official in Massachusetts asked “was the spirit of stateregulations for public involvement met?” which can only be answered subjectively. Problemsraised by subjectivity and definitional clarity are also apparent in another example. One metricsuggested was that “appropriate” response actions be used. However, the definition of what isappropriate is influenced on the type of oil spilled. For example, if the oil floats, there is lessneed to worry about it affecting, for example, nearby sunken ships (CP06). However, if the oilsinks those sites may be of more concern. “Appropriateness” can also depend on the kind ofartifacts or sites that are at risk. Chemical dispersants may not pose a risk to future research onthe site when its age is already known, but they may make it difficult or impossible to use carbondating for some sites (e.g., prehistoric sites). Similarly:
• They used fluidization where they super-saturate the ground to force the oil to thesurface and then clean it up…now archeologically, if you are looking at a stratified site, isthis going to ruin your stratigraphy? Is this going to force artifacts out of context so youend up with a layer with all your artifacts in it and you don’t have any stratigraphy(CP06).
Challenges related to measurability, such as reliability, validity, and accuracy, are discussedfurther in the following two sections.
6.3 What Makes a Good Performance Metric?As presented in our literature review report (and detailed in Graedel & Allenby 2002, Seager &Theis 2004) an ideal metric should be:
• scientifically verifiable. Two independent assessments would yield similar results.• cost-effective. It would use technology that is economically feasible and does not
require an intensive deployment of labor to track.• easy to communicate to a wide audience. People at large would understand the
scale and context and be able to interpret the metric with little additional explanation.• related to something that is important to many stakeholders. There is no point
assembling a metric no one cares about.
• related to something that can be changed by human intervention or would berelevant to human action. The metric would have a causal relationship between thestate of the system and the variables that are under a decision-maker’s control.Metrics that are independent of human action do not inform a management, policy-making, or design process.
• credible. It would be perceived by most of the stakeholders as accurately measuringwhat it is intended to measure.
• scalable over an appropriate time period and geographic region. It would beindicative of short, medium, and/or long term effects as appropriate. For example, itwould not be meaningful to attempt to measure the effects of chronic low-level toxicdosages over a period of weeks or months, just as it would not be appropriate toaverage local environmental conditions over a widely varying region.
• relevant. It would reflect the priorities of the public and other stakeholders andenhance the ability of spill managers and/or regulators to faithfully execute theirstewardship responsibilities.
• sensitive enough to capture the minimum meaningful level of change or make thesmallest distinctions that are still significant, and it would have uncertainty boundsthat are easy to communicate.
The interviews provide useful insights into the ways that different people involved with a specificspill response think about these issues; in fact, they identified many of these characteristics. Anunderstanding of how participants involved in spill response think about “good” characteristics iscritical for developing a process for collaboratively selecting metrics prior to a spill event.Stakeholders must develop clear criteria on which to assess alternative metrics; theirunderstandings can be informed by the literature on performance metrics (which we review inan earlier report) and by an understanding of their own views. In this section we will discusswhat we learned about peoples’ views of good performance metrics.
First, just because something can be measured does not mean it is relevant to understandingthe success of a spill response or important to many stakeholders. For example, severalinterviewees in both cases suggested that the amount of boom deployed could be used as ametric to assess how well a shoreline was protected from oil contamination. While it is easy tomeasure, it may not be a good indicator of whether the shoreline is actually impacted. As wewere told by many interviewees, deployed boom was not always effective when currents orwinds were strong. Another stated that “we tried to prevent oiling by putting out booming, butit’s not a very accurate predictor of whether the outcome [of shoreline protection] will beachieved. Once any oil gets on a beach you still need to clean it up.” Furthermore, deployingbooms may have been a good decision at the time it was made, but weather conditions can shiftand cause them to fail. That is, the metric does not have a causal relationship between thestate of the system and the variables that are under a decision-maker’s control.
Similarly, an interviewee suggested that the amount of money spent – for numbers of crew,manhours, and amount of boom deployed – may be very appealing politically as a measure ofperformance, but “is that doing the job to the most effective way?” and “there may be more costeffective ways to look at the whole picture. Money spent is not a good measure.”
As a third example, we learned that the amount of oil removed from beaches might be arelevant metric for assessing the response. In one sense this value might be easy to measure.But, some argued that the measurement is not meaningful: it is very difficult to quantify theactual amount of oil on the beach because it is mixed with sand and rocks and in sorbent
material: “Especially in recovered number you are getting a mixture of oil and water…skimmersnever perform as the manufacturers claim” (CP09).
Interviewees spoke of bird mortality and rehabilitation rates as an often used metric, but onethat might not always be that meaningful. For example, one person from the Chalk Point spillargued that the number of impacted swans is not a good measure because they are not nativespecies. The more relevant issue is the effort that was made to recover all the wildlife, not therehabilitation of those that have greater importance in the public’s eyes (CP03). In theBouchard case, we were told that:
• Does the recovery or rehabilitation of wildlife have any bearing on the effectiveness ofresponse…. NO, oil is in the water and it will hit certain number of creatures before weare even around and that has no impact on response…. There is an impact on how earlyyou get your wildlife rehab group involved; that’s were the correlation is – its not acorrelation to the response itself …the direct relationship is how soon have you calledout experts and set up triage and rehab centers? If you hesitate, you will have a highermortality rate and I have worked with a number of them that have gotten better andbetter at how they deal with injured wildlife (B10).
Second, criteria for gathering measurements must be well-defined and produce accuraterecording; they must be accurate, scientifically verifiable, consistent, and sensitive. However,this is frequently a problem. Different people may have different ideas about what is “impacted.”One way around this is training:
• We try to give them some training so that when they come back they say “we’ve got 200yards of shoreline heavily impacted”; what one calls “heavily impacted” is the samething… we try to get them to know what the shoreline types are so they can describe it inthe proper terminology (BI02)
Third, the causes of good or poor performance may be multiple and difficult to disentangle. Inother words, while measurement of a performance metric may suggest a poor response theresponse might have been “good” in the context in which the decision or action was made. Forexample, one person suggested that a relevant metric for assessing success is the amount ofoil recovered in open water before it hits the shore. However, in the Bouchard-120 spillresponse, very little was picked up in the open water because of 1) weather conditions and 2)poor coordination in the early stages. While the second reason might suggest a poorlyorganized response, the first reason is beyond the control of the responders; it would be unfairto judge the response as poorly implemented because of harsh weather conditions thatprevented a higher rate of offshore recovery. Similarly, in the Chalk Point spill there was aconcern about the amount of oil recovered or removed. Several interviewees thought that insitu burning might have been a better response option, but the time it would take to get all thepermits and necessary information in place precluded its use. The problem was not the lack ofcoordination, but rather other external factors limiting this option.
Fourth, what is most relevant may also be very difficult to measure. Many of the performancemetrics that could be easy to communicate to a wide audience, relate to something that isimportant to many stakeholders, changed by human intervention or would be relevant to humanaction, credible, and relevant are not easy to measure. That is, they would conflict with othercharacteristics of “good” metrics: are they scientifically verifiable and cost-effective?For example,
• Was there trust from the public?• What was the degree of change to beaches and sandbars from clean-up
actions?
Even measures that might at first glance seem easy to measure may not be, such as number oflost fishing days suggested by a Chalk Point interviewee: are fishing days “lost” due to impactsof the oil spill or because of foul weather or other reasons? Similarly,
• The IRAC sign-off generally happened a lot later than the actual clean-up wasfinished…in damage assessment we are looking at the impact over time…we want toknow how for long might birds be exposed [to an oiled beach] or how long did thatconcentration stay there so that maybe it chronically poisoned some shellfish…havingthat firm data point that we have confidence in prevents negotiation and discussion fromthe RP later…if we can firmly say, “this is when it was cleaned” versus a sign-off datebut it was cleaned a lot earlier than that that, it just provides uncertainty and needlessdiscussion (BI09).
• Initial walks with FWS when trying to pick up any injured or dead wildlife, I am sure a lotof natural mortality was captured in that particularly shad…it was natural mortality butthey were probably being tallied in as a result of the spill (CP11).
As another example, an endpoint for marshes could be no sheen or no oil available whentouched. However, oil coating may be left on peat or leaves because to address that mightresult in salt marsh destruction or further injury (BI10). Moreover, according to this individual:
• Its been proven that in oil contaminated salt marshes and reeds it is better just to leave itthere and it will rejuvenate. It will look bad the first year, but it’s gonna be just fine afterthat (BI10).
Similarly, when one person suggested that a successful clean-up is defined as no oil coming offto the touch, there is a problem of measurement arises. She argued that there is no way toquantify “not coming off to the touch” so its use engendered a lot of differences of opinion:
• I think we (IRAC teams) were pretty reasonable. Although the RP wanted to portray ourposition as if you have a mile stretch of shoreline and there’s one dot of oil that comesoff to the touch then that’s not good enough and we need to fail the segment. By nomeans were we that extreme at all… (BI07).
On the other hand, Similarly, qualitative metrics such as “no sheen’ was stated as a good metricby a state trustee in the Chalk Point case because, “you can send people out to do it [see if itsthere] and you don’t need any tools or equipment or training…sometimes the very crudemethods are best (CP04).
Fifth, having a clear threshold to differentiate “good” versus “bad” measures can be difficult todefine and is situation dependent:
• For rehabilitation [handling a lot of birds at one time] is yes [a successful measure]. itwould be a good metric. How fast were the teams able to set up the facility and werethey able to handle the capacity… [Survivability] depends a lot on the species involved,the type of oil spilled (BI09).
• I know the age, so I’m not too worried about them using chemical dispersants andruining my carbon dating… but it could be a problem with prehistoric sites (CP06).
Several of the interviewees discussed the difficulty of defining metrics with such characteristicsin relation to achieving clean-up goals in terms of end-points or outcomes. For example, inresponse to the oft heard question “how clean is clean?” one interviewee stated how it matterswho is asking: what is a threshold related to a metric?
• It depends on the habitat…we would like it cleaner usually than what they think issufficient (BI09).
It also depends on the species of concern. One person (BI08) suggested a list of possiblemetrics:
• For birds, it’s frequently that its unavailable, it doesn’t mean its not there, its just notgoing to rub off on feathers.
• These birds aren’t nesting…a little staining probably is not going to be a huge issuefor these guys and maybe the wildlife people might have a little bit lesser standard.
• If it’s in the fall, then we know that the marshes are starting to go into senescenceand this stuff will slough off and go away.
• If dealing with a recreational beach…the standard there is dependent on the user(human and bird) and the standard worked for both of them…so standard issmelling…of oil in sand.
These issues suggest that there are a number of important challenges to using performancemetrics to assess the quality of a spill response which meet the criteria for “good” metrics. Thechallenges identified by the interviewees are discussed next.
6.4 Challenges to using performance metricsA number of challenges to using performance metrics for assessing oil spill response can begleaned by the findings from this case study. Such challenges are important to document inorder to develop strategies for guiding performance metric selection in such a way that thechallenges are met. Pitfalls in performance metric selection and definition can be avoided by acareful understanding of these challenges; this is one of the features that our proposed processfor collaboratively selecting metrics prior to a spill event must overcome.
As can be observed by referring to Tables 4-22 there are an enormous number of potentialperformance metrics that can be applied to assessing the response effort. It is not realistic toexpect that all possible performance metrics should be used. There is a challenge ofdetermining which metrics are most relevant. And, this is complicated by the challenge createdbecause as interviewees put it:
• [Success] depends on where you sit. From my perspective, it’s ultimately what weprotect. How good we are at protection. That’s prediction, understanding having gooddata on the shorelines… closures on inlands and getting ahead of things. Its all basedon resource protection (BI08).
• It depends on how you define success. It depends on who is defining success…thereare different ways to measure it. Did we control it as best we could? Yes… We limitedthe impact…to areas that were ultimately, where the clean-up could occur (BI03).
• A lot of what we get to judge spills by is how the public feels about it after theresponse… public perception is our reality (CP03).
In fact, it is not just a challenge of sorting through possible metrics and choosing the “best”ones. There can be disagreements about what is a good metric for assessing performance.For example, one questioned the ability to objectively define “impacted”:
• [Number of miles of shoreline impacted as a measure] depends on the degree that itsimpacted, the type of shoreline, the environmental sensitivity of that shoreline, and thesocio-use of that shoreline…[also there can be impacts] from earlier spills… that wereunrecorded or not known about… coupled with remnants of oil from the B-120 spill(BI10).
On the other hand, a state agency representative stated that:• How many miles of shore line have been impacted? A valid measure of success;
probably the best you can come up with compared to number of miles that have beencleaned to an acceptable level. Typical measures that people want and can bemeasured
And, as presented earlier, the definition of what is appropriate is influenced on the type of oilspilled. For example, if the oil floats, there is less need to worry about it affecting, for example,nearby sunken ships (CP06). However, if the oil sinks those sites may be of more concern.“Appropriateness” can also depend on the kind of artifacts or sites that are at risk. Chemicaldispersants may not pose a risk to future research on the site when its age is already known,but they may make it difficult or impossible to use carbon dating for some sites (e.g., prehistoricsites). Already, we have presented several other exemplars of this problem, including questionsrelating to “how clean is clean,” oil “not coming off to the touch,” and the presence of sheen.
In addition, several other challenges were raised by the interviewees.
First, the issue of reliability of data recording was frequently mentioned in the interviews. Oneperson put it this way:
• Teams of trained observers go along the coastline and make observations of the extentand coverage area of oil. They try to record that in a unified way that does not have a lotof variability in it… [They are basically recording: 1) where’s the oil, 2) something abouthow much, and 3) what the shorelines are like (BI02).
• When the SCAT teams…go through they pick up rack and oiled things and they are notalways discriminating if this is a dead bird or not. We highly suspect that there are deadbirds getting scooped up put in the trash bag with all the other oiled stuff…withoutgetting documented…if there was a way to…have that separated out…that’s anoption…there ends up being a lot of data gaps (BI09).
• If there’s one thing that would help me a lot is to…make sure that since both responseand NRD are getting there hands on the same types of data at least if we are collecting itin a way that’s going to be consistent and helpful to both sides I know that there issomeone responsible in incident command for collecting all the data…but what I am notsure happens is that someone looks at the forms and provides some feedback orcorrections to the SCAT teams themselves as encouragement to make sure all theblanks are filled out, etc...Spaces on the form are not always filled out and that doesn’thelp me (BI09).
Second, there is little accurate baseline data on which to base comparisons or assessments.For example,
• We can clean the beach, but if we lose an environmentally sensitive area, like a bigshellfish bed, that’s a lot harder to replace so lets pick our priorities…There wasn’t anestablished criterion of how many ppm of hydrocarbons in shellfish is dangerous…Thereare spills here from these boats all summer long…each municipality needs to know whatthe baseline is and what the safe baseline is…what’s allowable and what’s standard?(BI05).
Third, measurements and assessments may evolve overtime. For example, clarification of whatis meant by “heavily impacted.” One interviewee said:
• Getting the shoreline reasonably back to where it should be…it was a subjectivejudgment call with each section of the shoreline based on the criteria and adhering to thecriteria… We refined the process as we went along over time because you have more of
a sense of perspective after you’ve seen a bunch of different segments of the shore-line(BI07).
Fourth, metrics can measure performance on multiple dimensions. When looking at the % ofincrease in organic matter over time in a newly constructed wetland, this was said:
• Sometimes it takes 7 or 8 years for wetland to show organic matter increase… it’sextremely variable so we couldn’t use that as a metric. But we know that that’s whathappens in a wetland and if it doesn’t, than something’s haywire (CP07).
Fifth, it can be very hard to define metrics so that they are measurable. It is not always clearwhat kind of data can reasonably be gathered to measure some of the metrics suggested.Many of the metrics suggested by the interviewees can be measured quantitatively, such asgallons spilled, number of birds injured or released, amount of money spent, number of peopleinvolved in cleaning beaches, etc. Of course, there may be uncertainties associated with themeasured values, but in principle they can be measured quantitatively.
On the other hand, a large number of the performance metrics identified by the interviewees donot lend themselves to quantitative measures. Some may be measured simply “yes” or “no”:was there conflict among staff in the command center? However, such measures do not assessthe significance of the conflict, and, hence, may not be very informative. Other metrics can onlybe assessed qualitatively:
• Are members of the public happy?• Was there trust from the public?• Are response actions having the desired effect?• Good working relationships with all parties involved?• Shoreline back to conditions prior to spill?
Sixth, performance metrics that some federal and state responders think are relevant can bevery difficult to communicate to the public – or at least that is their perception. This issue wasraised repeatedly in regard to the impacts of clean-up efforts. Many interviewees from bothcase studies provided examples of when, according to them, it would have been better to ceaseclean-up actions because more severe impacts would result than if none or no further actionswere taken. Their ability to articulate these trade-offs to the public were difficult. Similarly:
• We took sample after sample after sample [testing for presence of PHCs in fish] andeverything was fine...the only problem was convincing people that everything was fine(CP03).
And, as a final example, local residents observe clean-up crews. But they might not fullyunderstand what they are doing, and reach incorrect conclusions:
• There were hundreds of people (workers) here working…some of them leaned on theirshovels quite a bit of the time (CP15).
• People were reporting that clean-up crews being negligent, because of throwing rocksback into he water. But that was what they were instructed to do after wiping them off –and here is the rationale for doing that. After we explained this to them they wereaccepting of that process (BI11)
Seventh, when a measurement is taken it can affect characterization of performance on ametric. For example,
• Cleanup crews would go down [to a beach] and work all day long (raking and bagging)…the next morning it looked like they had done nothing. So the oil was coming fromsomewhere how much was still sort of stored offshore we never knew (BI03).
• You have to be on auto-pilot…It takes time for people to get there so you have to let theContingency Plan run on autopilot for that first 24 hours (BI09).
• It [signoff] identifies the termination; it doesn’t necessarily say we are successful, it justsays we’re done; It doesn’t say we did the right thing, it just says we’re done; that’s yourticket home (BI08).
Finally, many of the performance metrics suggested are indirect measures of something that theinterviewees really cared about. They are not direct measures of, for example, economicimpacts, public satisfaction and support, public and worker health, or ecological impacts:
• A lot of oil was removed from the environment by the teams that were out therespreading sorbent material and scraping rocks…most of that material was less of anenvironmental threat as it was an aesthetic threat to property owners (CP04).
Similarly, many end-points suggested are rarely directly about the health or quality of theecological system or its components. In some cases the metrics assess indirect or interimindicators of the state of an ecological system, habitat, or populations. For example:
• How many pounds of debris, oil contaminated debris, has been recovered and sent offfor disposal? (BI02)
• Number of miles of shoreline impacted depends on the degree that its impacted, thetype of shoreline, the environmental sensitivity of that shoreline, and the socio-use ofthat shoreline (BI10)
• How many birds were oiled, how many recovered, how many released, and how manyreleased and survived? (BI02)
• How many acres of shellfish areas were closed and then re-opened? (BI02)
These challenges are certainly not unique to the assessment of oil spill response. Problemsassociated with measurement and the lack of baseline data are widely documented. Problemsassociated with complex, dynamic systems for which longterm outcomes are of interest but verydifficult to assess have been discussed in the context of international conflict resolution(National Research Council 2000a) and longterm stewardship (National Research Council2000b). For example (National Research Council 2000a, pg. 63):
We have noted that evaluation is complicated by the fact that short-and long-termdefinitions of success may be quite different. This difficulty can be addressed in part byfocusing on particular outcomes rather than overall “success.” Evaluations can beseparated according to time horizon, with outcomes at different times analyzedseparately. As noted, it is important to have short-term indicators of progress even forinterventions intended mainly to have long-term effects. This is so partly to provideinterim indications of progress and also to allow for meaningful evaluation even in casesin which intervening events not brought about by the intervention throw the process ofconflict resolution off its intended course.
One way to develop interim indicators of progress for interventions that have a long timehorizon is to postulate mechanisms believed to lead from the intervention to the desiredlong-term results and identify indicators that can be used to tell whether events aremoving along the desired track. For instance, practitioners of interactive problem-solvingworkshops as a way to improve intergroup communication over the long term mightpostulate that one mechanism leading to long-term effects such as formal agreementsbetween the parties is through improved communication and trust between workshopparticipants from opposing sides and their advancement over time to more influentialpositions in their groups. One could assess whether this mechanism is operating byexamining various indicators, such as improved communication between the participants
immediately after the workshop, continued communication between them a year or twolater, the rise of workshop participants to positions of increased influence, and evidenceof sensitivity to and accommodation of the opposite group in the policies they influence.Using such indicators of intermediate progress improves on an outcomes-only approachto evaluation by adding a process element to the evaluation and by strengthening thecase for a causal link between the intervention and long-term outcomes.
6.5 Disentangling metrics for preparedness, response, damage assessmentWe found that some people assessed the response to the Bouchard-120 and Chalk Point spillson the basis of how well preparations were made prior to the response. Their concern forpreparedness was related to for a) gaining public support for the response and b) ensuringeffective response effort, coordination, etc.
For example, the following were suggested as metrics to assess the response to the spill:• Were possible “stop gap” areas for booming pre-identified?• Measure of place-specific baseline and definition of safe levels available?• Established ppm standards for toxicity of hydrocarbons that are harmful to shellfish?• Previous workshops and contingency planning on specific species (roseate terns) and
sensitive areas (Ram Island) conducted?• A plan about end-points to address that is time-based developed ahead of time?• Training of local and state responders in Incident Command (ICS) conducted prior to
spill event?• Pre-selected staging areas identified and prior approval from landowners obtained?• Prior notification plan set in place?• Designation of official volunteer coordinator prior to spill?• Clean-up endpoints predefined in response plans, based on risk of direct oiling?• Established phone numbers for people to call and cell phones, etc. are available• People able to contact each other using available communication equipment?• Protocols established to reopen a closed fishery prior to spill event?• Booms available in local communities prior to spill event?• Key players are pre-identified?• Safety training for clean-up crews conducted?• Contingency plans that identify priority sites developed that are up-to-date? (e.g.,
location of cultural artifacts)• Training for state officials about state rights and responsibilities and the process of oil
spill response and damage assessment• Emergency plans set in place• Responders received a primer on what to expect and what will happen now and who is
responsible for what and how will communications take place• Number of pre-trained people in case spill happens again• Number monitors on pipeline• Enough spill equipment available• Number of table top exercises/ drills• There is an organized system of getting the information out.
Furthermore, some of the interviewees assessed the quality of the response based on how wellit supported the needs of damage assessment and restoration related activities. For example,several interviewees in both case studies stated that an important performance metric for theresponse effort was how well it developed the data necessary for damage assessment:
• Successes for NRDA is, good things that should happen for NRDA are thingslike…making sure we get all the data collected. And that would take some thought onhow to put a performance metric on to that. It’s a tough one (BI09).
• Maybe some other water samples but more strategic…maybe a more strategiccollection…to basically validate the model. We used [her SIMAP] model to projectimpact to aquatic resources…it predicts water column concentrations of petroleum. Ifyou could use a few samples to confirm that what its predicting is sort of what we areseeing from dissolved concentrations I think that would have been useful…you’ll neverbe able to model it appropriately but it’s a good tool (CP01).
These findings suggest that people do not necessarily consider “response” as distinct frompreparation, damage assessment, and restoration. In fact, these activities can overlap in time.Activities considered as part of the response can extend for many years after the spill event andthe initial clean-up. Such is the case in the Bouchard-120 case, where several years after thespill occurred some activities related to “response” are still being conducted.
7.0 ConclusionThe case studies of the Bouchard-120 and Chalk Point spill responses suggests that interestedand affected parties can have multiple objectives. They may also use multiple measures forassessing achievement of or progress on those objectives. By and large, they were cognizantof the usefulness of assessing response efforts both during and after a spill response.
However, these case studies also illustrates that objectives may not be entirely shared amonginterested and affected parties. People involved with a particular spill response may havedifferent objectives and prioritize them differently. Most strikingly, the desire to address theneeds and concerns of the affected public/communities was expressed almost entirely byfederal and state officials and a contractor for the responsible party – but not by local officials orthose who worked for NGOs. This may be a reflection of their concern for obtaining the trustand acceptance of the public. They want their response actions and achievements to beconsidered legitimate and reasonable. This finding may also be an artifact of our sample: wedid not interview residents of the communities affected by the spill and the response.
There may also be different objectives that have salience to responders in different spills. Forexample, protection of cultural artifacts was only raised in the Chalk Point case, and a concernfor mitigating social nuisance impacts was raised only by interviewees in the Bouchard-120case.
Our interview samples were intentionally drawn from people that were actively engaged in thetwo spill responses. This also limits what we can say about potential interested and affectedparties that did not participate actively – which includes much of the local communities affectedby the spills and responses. While we did not interview many local residents for these casestudies (because we intentionally selected people who were engaged with the response), anumber of people spoke of their experiences with members of the public. In particular, localresidents were described as wanting to know very practical information that related to impactsand response: what were clean-up schedules? What were public health effects of consumingfish contaminated (or that might be contaminated) by oil? In fact, some of our intervieweescharacterized members of the public as wanting things that were impossible or that were notreasonable: e.g., remove all oil. While we have no doubt to question the reasonableness offederal and state level interviewees, we think it is important to question the validity of theirdescriptions and attributions about local residents. In fact, a few of those that we interviewed
were quite sophisticated in their views. We expect that further tasks associated with thisresearch project will explore such issues in more depth (e.g., masters thesis of Rebecca Kay).
We also found that while respondents think performance metrics are important and useful, theremay be differences about when they should be developed and used. For example, responseperformance metrics were developed by the Unified Command during the Bouchard-120 spill:
• how do you guarantee that we are meeting our objectives? How do we measuresuccess? We sat down as a group – three weeks into the spill – because first threeweeks we are just responding…We ask the bird wildlife and rehabilitation people, whatare your success factors? They said we have a goal of 20% rehabilitated. We said OK,that is the objective of the bird people. We asked the safety people, what is yourobjective? They said, occupational safety and health said this many hours, no mishaps.So we started tracking that number. I delegated to a planner the need to work withthese kinds of groups and develop metrics and put them on a poster so people couldtrack them….
The underlying assumption of our research project is that better oil spill response can beachieved if performance metrics are defined prior to a spill in a systematic and collaborativeprocess. Spill response metrics can drive which strategies are chosen. Spill response metricscan also help officials gain the legitimacy and trust they hope for – if they can show that theirefforts measure well on the chosen metrics. Spill response metrics can also drive learning.
These case studies and their analysis are an important, but initial step in addressing ourresearch goals. Specifically, while we designed these case studies to better understandobjectives and performance metrics, they were not designed to elicit relative priorities aboutobjectives or specific “good” metrics to measure them in a systematic manner. These tasks arepart of the next step of our project, which must be informed by the results of these case studies.A better understanding of how objectives are prioritized and how priorities differ amongstakeholders is an important component of creating a collaborative proactive approach todeveloping oil spill response metrics. In other words, the process will seek to lay bare the valuejudgments about which some of our interviewees spoke:
• Success depends on how you define it; it depends on who defines it and how they defineit (BI03).
• Termination endpoints depend “on whose ox is being gored” (BI08).
ReferencesGlaser, B. and Strauss, A. 1967. The discovery of grounded theory: Strategies for qualitative
research. Chicago: Aldine.
McCaffrey, 2002
Moore, S. A. 1996. Defining 'successful' environmental dispute resolution: case studies frompublic land planning in the United States and Australia, Environmental Impact AssessmentReview 16:151-169.
Tuler, S. and Webler, T. 1999. Voices from the forest: What participants expect of a publicparticipation process. Society and Natural Resources 12:437-453.
Glaser, B. 1992. Basics of Grounded Theory. Mill Valley, CA: Sociology Press.
Seager TP, Theis TL. 2004. A taxonomy of metrics for testing the industrial ecology hypothesesand application to design of freezer insulation. Journal of Cleaner Production 12(8-10):865-875.
National Research Council 2000a. International Conflict Resolution After the Cold War.Washington, DC: National Academy Press.
National Research Council 2000b. Longterm institutional management of US Department ofEnergy Legacy Waste Sites. Washington, DC: National Academy Press.
Appendix A: Interview guide
A. IntroductionGive overview of our research project and goals and generally what we want to learn about. What willhappen to the information they tell us. Consent form.
B. Roles and experiencesGoal of questions: gather information about how each person was part of the spill and response;generally ease them into talking about their memories.
1. How were you involved with the Bouchard spill and response?
Possible probes:What was your role? What were your responsibilities?Who else did you interact with as part of the spill response?Did you have pre-defined procedures or activities (e.g., response plans)?Were these followed?
C. Concerns about the spill and responseGoal of questions: gather information about each person’s concerns related to the spill and response. Inaddition, to understand if they had any organizational concerns (similar or different).
2. When the spill happened, what were you most worried about?3. As spill response activities unfolded, did your worries change? If so, how?4. Do you still have worries, in the aftermath of the spill response? What are they?
Possible probes:How were you personally (and family) affected by the spill and response?How was your community affected by the spill and response?How was your organization or business affected by the spill and response?What was the most important response need? (e.g., recover oil? Reduce impact to scallop beds?)What do you think that the federal agencies were most concerned about?What do you think that the state was most worried about?What do you think that local community members were most worried about?
D. Views about the responseGoal: begin to gather information about how spill response quality is measured.
Do you think that the spill response was a success or failure? Why?What information has been helpful for you to determine whether the response is complete and was asuccess? Who was the source of that information? Did the information address your worries?Did any of the response activities have much of an impact on ameliorating the impacts of the spill?
Possible probes:When you consider whether the response was successful (or not), what comes to mind?What measures of success or performance in response have been used by different stakeholders andorganizations? By the federal agencies (NOAA, Coast Guard)? By others? By NGOs (e.g.,, BuzzardsBay Coalition)? In legal suits?Do you know if these measures have been used in other recent spill response efforts?Did any of the response activities make things worse? If so, how?Do you think that they responsible agencies were adequately prepared for the spill and its response?Why?Was spill response “by the book”? Were their any problems, e.g., organizational jurisdiction? Resourcesavailable and devoted to response?
E. Views about response performance metricsGoal: get feedback about whether folks thought that the metrics used to explain spill responseperformance and impacts/risks were relevant, useful, etc.
What information would you have liked in order to assess the quality of resources/ecosystem (services)that were impacted by the spill?What information would you have liked in order to assess the quality of the response? [NOTE: this mightlead them to talk about job performance issues, but as I think about it, if that is what they want to tell us isimportant to them, then that is worth eliciting! In any case, that is why I am putting it second.]Were their concerns/impacts/risks that were not well characterized by the measures used to judge theresponse?
Possible probes:Here the interviewer needs to keep track of a) what were stated as concerns and b) what kinds ofmeasures were identified to measure success/failure. Iterate through them. Make sure to inquire abouteach of the concerns – explore whether there were even measures related to them.
Do you think that these measures provided a good way of characterizing the impacts you were worriedabout? Why?What information would be best to measure or characterize the things you are concerned about? [giveexamples for them?]
